September 22, 2026

Most of the frontier open-weight models released in the last two years, Mixtral, DeepSeek-V3, Qwen3-MoE, Kimi K2, share an architectural choice that is easy to skip past in a release announcement: they are not dense transformers. They are Mixture of Experts (MoE) models. The distinction matters because it changes what “number of parameters” means, and it explains why a model can list hundreds of billions of parameters while running inference at a cost closer to a model a tenth of that size.

The problem MoE solves

In a dense transformer, every parameter in every feed-forward layer is used for every token. Making the model bigger means more parameters and proportionally more compute (FLOPs) per token, at both training and inference time. Capacity and compute cost are locked together.

MoE breaks that link. The feed-forward block in each transformer layer is replaced with several parallel feed-forward networks, called experts, plus a small router that picks a subset of them for each token. Only the selected experts run; the rest stay idle for that token. Total parameter count grows with the number of experts, but compute per token stays tied to how many experts are active, not how many exist. This is what “total parameters” versus “active parameters” refers to when people describe an MoE model.

Attention layers are usually left dense. Only the feed-forward layers are made sparse, since that is where most of a transformer’s parameters live.

How routing works

For a token with hidden state x, a gating network (a single linear layer in most implementations) produces a score per expert: scores = softmax(W_g · x), one value per expert.

The router keeps the top k experts by score (top-2 in Mixtral, top-8 among 256 routed experts in DeepSeek-V3, top-1 in the original Switch Transformer) and routes the token to only those. The output is the weighted sum of what the selected experts produce, using their own scores as weights. A minimal version of the forward pass looks like this:

def moe_layer(x, experts, gate, k):
    scores = softmax(gate(x))          # one score per expert
    top_idx = topk(scores, k).indices  # which experts this token uses
    out = 0
    for i in top_idx:
        out += scores[i] * experts[i](x)
    return out

Everything else, attention, normalization, residual connections, is unchanged from a dense transformer. MoE is a drop-in replacement for the feed-forward block.

Load balancing

Left on its own, training pushes the router toward a small set of experts that happen to get an early advantage, a failure mode usually called routing collapse: those experts keep receiving gradient updates and improving, the rest are undertrained, and the extra parameters go to waste.

The original fix, used in GShard and the Switch Transformer, is an auxiliary load-balancing loss added to the training objective, penalizing the model when tokens are distributed unevenly across experts. It works but competes with the main loss, and getting the weighting wrong can hurt model quality.

DeepSeek-V3’s technical report describes a different approach: an auxiliary-loss-free strategy where each expert has a bias term added to its routing score. After each batch, the bias is nudged up for experts that received too few tokens and down for experts that received too many. This adjusts routing without adding a competing gradient signal to the loss.

Shared experts

DeepSeek-V2 and DeepSeek-V3, followed by Qwen3-MoE, add one or more shared experts that process every token in addition to the routed ones. The idea is to separate general knowledge, handled by the always-on shared expert, from more specialized knowledge, handled by whichever routed experts the gate selects. It also gives the router less work to do, since common patterns do not need to be re-learned redundantly across routed experts.

DeepSeek-V3 also uses many more, smaller experts than earlier designs (256 routed experts, 8 active per token, plus 1 shared expert) instead of a handful of large ones. Finer granularity gives the router more combinations to choose from for a similar active-parameter budget.

What it costs

MoE is not a free way to get more capacity.

• Memory. All experts have to be loaded, since routing decisions are made per token and any expert can be selected at any time. A model with 671 billion total parameters needs enough memory (across GPUs, in practice) to hold 671 billion parameters, even though a single token only touches 37 billion of them.
• Distributed training and inference. Experts are usually spread across GPUs (expert parallelism), and each token’s hidden state has to be sent to whichever GPU holds the expert it was routed to, then the result sent back. That all-to-all communication is a real bottleneck, and it is one of the reasons DeepSeek-V3’s report spends as much space on communication engineering as on the model architecture itself.
• Batching. In a dense model every token in a batch takes the same path. In an MoE model, tokens in the same batch are routed to different experts, so an efficient implementation needs a way to group tokens by expert before running them, and to handle experts that end up with more or fewer tokens than expected.

Known models and their numbers

(Kimi K2’s architecture was covered in an earlier post.) GPT-4 has never been officially confirmed as an MoE model, but it was reported to be one in June 2023, with 16 experts of about 111 billion parameters each and 2 active per forward pass, in a leak analyzed by SemiAnalysis that OpenAI never confirmed or denied.

Summary

MoE replaces a dense feed-forward block with several experts and a router that activates only a subset per token, decoupling total parameter count from per-token compute. The open questions it introduces, how to balance load across experts without hurting training, how to place experts across GPUs, how to batch tokens efficiently, are less about the architecture’s core idea and more about the engineering needed to make it run efficiently at scale. That engineering effort is a large part of why recent MoE technical reports (DeepSeek-V3’s in particular) read as much like systems papers as machine learning papers.