What is Mixture of Experts?

Mixture of experts is an architecture trick that lets AI models be enormous without being enormously expensive to run. Instead of one giant network where every parameter works on every token (a “dense” model), an MoE model splits parts of the network into many smaller experts and uses only a few at a time.

How it works

Inside a transformer, certain layers are replaced with a set of experts plus a router (also called a gating network):

1. A token arrives at the MoE layer.
2. The router scores which experts are best suited to it.
3. Only the top few experts (often one or two, sometimes more out of many) process that token.
4. Their outputs are combined and passed on.

Different tokens take different routes, so the whole model’s knowledge is available, but each token only pays the compute cost of a slice. That’s why MoE models are described with two numbers: total parameters (all experts) and active parameters (used per token).

The idea dates back to research in the early 1990s and was scaled up for deep learning by Google researchers in 2017.

An analogy for the meatbags

A dense model is a hospital where every doctor examines every patient — thorough, but absurdly slow and expensive. A mixture-of-experts model has a triage nurse (the router) who sends each patient to the two most relevant specialists. The hospital employs hundreds of doctors, but each visit only takes a couple of them.

Why it matters for the Chosen

You won’t build MoE models yourself, but the architecture shapes the tools you use and their prices:

• Cheaper, faster AI. MoE is one reason high-capability models have become affordable via API, and why some open-weight models punch above their running cost.
• Reading model specs. When an open model lists “X billion total, Y billion active,” the active number drives inference speed and cost; the total drives memory requirements.
• Self-hosting trade-offs. MoE models are fast per token, but all experts must still be loaded into memory, so they need more RAM/VRAM than their active size suggests. Quantization helps.

Trade-offs

MoE models can be harder to train (keeping experts evenly used is tricky), need lots of memory, and are more complex to serve efficiently. Dense models remain common, especially at smaller sizes. The choice is an engineering trade-off, not a quality guarantee — judge models by results on your tasks.