What is a mixture of experts (MoE)?

A mixture-of-experts (MoE) model splits parts of the network into many parallel sub-networks — "experts" — and adds a router that picks a small number of them for each token. The rest sit idle.

The result is a model with a huge parameter count that runs at the speed of a much smaller one.

Total vs. active parameters

This is the only thing you really need to take away. An MoE model has two different sizes, and marketing quotes the flattering one:

• Total parameters — everything in the file. Determines how much memory you need to load it.
• Active parameters — what actually runs per token. Determines how fast it is and what it costs.

A model might have eight experts but use two per token, so roughly a quarter of it works at any moment. You get the knowledge of the big number and the speed of the small one.

The catch is that "sits idle" is not "isn't there". Every expert must be loaded in memory, because the router might pick any of them for the next token. You pay for total parameters in RAM and active parameters in compute.

Why that trade is attractive

Dense models — where every parameter runs for every token — have an unforgiving scaling curve: more capable means proportionally slower and more expensive, always.

MoE breaks the link. Add experts and the model knows more without each token costing more. That's why the technique, which dates to 2017 and was demonstrated convincingly at scale by Mixtral in 2024, is now widespread among frontier models.

What it means depending on who you are

Using a hosted model: essentially nothing. It's an implementation detail that shows up as better speed-per-price. You cannot tell from the outside, and you shouldn't need to.

Running models locally: it matters a lot, and it cuts against you. An MoE model needs memory for all its experts while giving you the compute cost of a few. On a machine where memory is the constraint — which is every consumer machine — a dense model of the same memory footprint may serve you better. Check both numbers before downloading 90GB.

Reading announcements: when a company touts a huge parameter count, look for the active count. The difference between the two is often the difference between the headline and the product.