QRoute

A variational quantum circuit that routes tokens to experts in a Mixture-of-Experts LLM.

---

A Mixture-of-Experts layer routes each token to a few of many expert FFNs. The router is a tiny classifier — and a natural place to ask whether a quantum circuit can do the job, because n qubits give 2^n computational basis states, one per expert, so a small circuit can score an exponential number of experts. QRoute prototypes this: a 3-qubit VQC routing top-2 of 8 experts on a toy MoE, with a classical baseline, and a publishable plan to scale to a 7-qubit router over 128 experts in Gemma 4.

Try it now

Open the live demo on Hugging Face → huggingface.co/spaces/Laborator/qroute

Pick an example token and watch the 8 expert "lamps" light up — which experts the VQC router and the classical baseline each select — with routing-probability bars, training-loss curves, and expert load balance.

from qroute.training.train import train_toy_moe

metrics = train_toy_moe("quantum")   # or "classical"
print(metrics["val_acc"], metrics["router_params"])

What is this?

When an LLM scales with Mixture-of-Experts, each token is routed to a small subset of expert networks by a learned router. QRoute asks: can that router be a quantum circuit? The appeal is structural — n qubits produce 2^n measurement outcomes, so 7 qubits could score 128 experts while the circuit's parameter count grows only with its depth, not the number of experts.

This repository is Phase 1: a working prototype on a toy MoE (8 experts, top-2, 3 qubits), trained end to end with PyTorch on the PennyLane simulator, with a classical MLP router for comparison. It is a feasibility demonstration and a foundation for the full-scale design — not a Gemma integration and not a claim of quantum advantage.

How it works (in 30 seconds)

• A learned linear layer maps a token embedding to 2 x n_qubits rotation angles.
• A VQC (data encoding + variational layers + a CNOT ring) is measured with qml.probs, giving 2^n_qubits expert routing scores from n_qubits qubits.
• Gumbel-softmax top-k selects the experts (differentiable in training, hard at inference).
• The MoE output is the routing-weighted sum of the selected experts.

More detail

The toy MoE trains router + experts jointly on a synthetic 8-cluster classification task. The VQC router has 12 variational parameters (p=2 layers, 3 qubits) plus the encoding projection; the classical baseline is a small MLP at a comparable count. See docs/vqc-router-explained.md.

Toy results

8 experts, top-2 routing, synthetic 8-cluster task, 40 epochs, seed 0. Reproduce with python scripts/train_toy.py --router both.

Router	Params	Val acc	Train acc	Expert Gini
VQC (3-qubit)	114	100%	100%	0.246
classical (MLP)	108	100%	100%	0.526

Feasibility, not advantage. Both routers converge to the same accuracy at a comparable parameter count; the VQC router is a trainable, integrable quantum module, not a faster or more accurate one. The toy task is easy enough that both solve it. On this run the VQC happens to balance expert load more evenly (lower Gini), which is an observation, not a claim.

Architecture

flowchart LR
  T["token embedding"] --> E["learned encoding<br/>-> 2n angles"]
  E --> V["VQC<br/>encode + variational + CNOT ring"]
  V --> M["measure (qml.probs)<br/>2^n expert scores"]
  M --> K["Gumbel-softmax top-k"]
  K --> X["weighted sum of<br/>selected experts"]
  T --> C["classical MLP router"]
  C --> K
  X --> O["MoE output"]

Loading

Full-scale design

The full-scale plan — a 7-qubit / 128-expert / top-8 router for Gemma 4 26B, with the encoding, ansatz, DeepSpeed training plan, and IBM Heron validation — is in docs/design.md, with an explicit "not yet implemented" section. Phase 1 (this repo) is done; Phases 2-4 are planned, not built.

Roadmap

docs/roadmap.md lays out Phase 2 (scale the router to 7 qubits / 128 experts), Phase 3 (Gemma integration + DeepSpeed), and Phase 4 (IBM hardware validation + write-up), with effort estimates and open research questions.

Limits (honest scope)

• Simulator only, 3 qubits / 8 experts. No real quantum hardware.
• Toy task. Synthetic clusters, easy enough that the classical baseline also hits 100%; this shows feasibility, not advantage.
• Not a Gemma integration. No real model is loaded; the full-scale design is a plan, not running code.
• No claim of quantum advantage. The value is the architecture and a trainable prototype.

Citation

@misc{brinza2026qroute,
  author       = {Serghei Brinza},
  title        = {QRoute: a variational quantum router for mixture-of-experts LLMs},
  year         = {2026},
  publisher    = {GitHub},
  howpublished = {\url{https://github.com/Quantum-Labor/qroute}},
}

The Quantum Co-Processor program

Three projects exploring quantum subroutines behind classical LLM interfaces:

	Project	What	Status
1	QVerify	Grover-assisted verification of LLM reasoning	shipped (Space)
2	QAgent	QAOA tool selection	v0.2 (Space)
3	QRoute	VQC mixture-of-experts router (this repo)	Phase 1 (Space)

License

Apache 2.0. See LICENSE.

Author

Serghei Brinza (@SergheiBrinza)