Key takeaway: A mixture-of-experts (MoE) model routes each token to only a few of its expert FFNs. DeepSeek-V3 activates 37B of its 671B parameters per token. That's cheap compute, not cheap memory: routing is decided per token, at inference time, so every expert must stay resident in GPU memory whether it works this step or not. MoE pays off only when sustained volume is high enough to amortize that fixed floor.
Every prior post in this series assumed a dense model, where every parameter works on every token. MoE is the upstream architecture call: FLOPs scale with active parameters, VRAM scales with total parameters, and the two can differ by more than 10x.
Previous post: Tensor vs Pipeline Parallelism: Splitting Models Across GPUs, the two axes expert parallelism becomes a third one of.
What this post covers
- What MoE is, mechanistically: router, top-k selection, and where it came from
- Why MoE inverts the usual compute/memory tradeoff
- Real total-vs-active parameter numbers across four production models
- What it does to serving: expert parallelism, the all-to-all tax, fragmented batches
- Which serving engines support it today
- The business "so what": when the trade pays off, and when it doesn't
Mixture of experts, mechanistically
MoE replaces one sublayer (the feed-forward network, or FFN) inside a transformer block; attention stays dense. Each layer is a router (scores every expert per token) plus N expert FFNs. The router picks the top-k highest-scoring experts and weights and sums their outputs. The rest do zero work: sparse activation, decided fresh per token, per layer.
It traces to Shazeer et al.'s 2017 conditional-computation work on LSTMs, a claimed 1000x capacity gain at near-constant compute (arXiv:1701.06538), later refined by Switch Transformer's top-1 routing, which trained a trillion-parameter model 4-7x faster than dense T5 at equal quality (arXiv:2101.03961). It's now the default for frontier open-weight models: DeepSeek-V3, Mixtral, Qwen3, and GPT-OSS are all sparse.
Why the compute/memory tradeoff inverts
A dense model's FLOPs per token scale with all its parameters; MoE's scale with only the active subset. That's the pitch. Then memory enters the picture:
- Data-dependent routing. Routing is decided per token, at inference time, so nothing tells you in advance which expert it needs. Nothing can be pre-loaded or safely evicted: every expert stays resident, and VRAM is sized to total parameters, not active ones.
- A regime-dependent compute edge. Epoch AI finds an 8-way-sparse model performs like a dense model roughly half its active size, but only below roughly 300 concurrent requests on an H100-class GPU, while decode stays memory-bandwidth-bound. Past that threshold, the edge narrows. Communication is the cleaner win: roughly 50% less inter-GPU traffic than an equivalent dense model at the same FLOP budget (Epoch AI).
The real numbers: total vs. active parameters
- Mixtral 8x7B: 46.7B total, 12.9B active, 8 experts, top-2. Mistral's own line: it "processes input... at the same speed and cost as a 12.9B model" (Mistral AI).
- DeepSeek-V3: 671B total, 37B active, 256 routed experts plus 1 shared, top-8 routed (arXiv:2412.19437).
- Qwen3-235B-A22B: 235B total, 22B active, 128 experts, top-8, no shared experts (a change from Qwen2.5-MoE) (arXiv:2505.09388).
- GPT-OSS-120B: 116.8B total, 5.1B active, 128 experts per layer across 36 layers, top-4 routing. OpenAI's own model card says MXFP4-quantized MoE weights let it fit on a single 80GB GPU (arXiv:2508.10925).
What this does to serving
- Expert parallelism (EP) is a third parallelism axis, distinct from tensor and pipeline: it shards experts, not layers or heads, once expert weight exceeds one GPU's memory.
- An all-to-all tax at every layer. EP needs an all-to-all exchange at every MoE layer: tokens route to whichever GPU holds their expert and back. It doesn't scale past hundreds of devices. Latency grows roughly linearly with device count, and the group stalls on whoever holds the most-routed-to expert (DeepSpeed-MoE, arXiv:2201.05596; survey, arXiv:2412.14219). DeepSeek-V3's auxiliary-loss-free balancing, a dynamic per-expert bias, is one production fix (arXiv:2408.15664).
- Batches fragment per expert. Per-expert batch size can stay small even when the aggregate is large, undercutting continuous batching's gains. Expert GEMM kernels need roughly 256-512 tokens/batch to hit their throughput knee point; halving that can nearly double expert latency (arXiv:2512.09277).
Which serving engines actually support this today
- vLLM: native Expert Parallel Deployment support for DeepSeek-family MoE, with disaggregated prefill/decode.
- SGLang: large-scale EP plus DeepEP, DeepGEMM, and DeepSeek's EPLB to correct routing skew. LMSYS reported 52.3k input and 22.3k output tokens/s per node serving DeepSeek across 96 H100s (LMSYS).
- TensorRT-LLM: Expert Parallelism plus "Wide-EP," built for GB200 NVL72's 130 TB/s NVLink domain to offset the all-to-all tax (NVIDIA).
The business so what: MoE pays off at volume, not at burst
Mistral pitched Mixtral 8x7B as matching or beating Llama 2 70B quality "at 6x faster inference." That claim is real, but it assumes traffic dense enough to keep a GPU holding 46.7B worth of weights continuously busy.
- High, sustained volume wins, amortizing the fixed VRAM floor and keeping per-expert batches near the GPU's efficiency knee-point.
- Low or bursty volume loses on both axes: full memory cost (671B-worth, for DeepSeek-V3's 37B-worth of compute) without enough tokens to amortize the floor or hit efficient per-expert batch sizes.
That's a serving-layer call, the same axis as continuous batching and KV cache sizing. It says nothing about whether the tokens flowing through those experts are grounded in the right context. That's the context layer, and no expert parallelism fixes a stale retrieval.
The bottom line, and where to start
MoE isn't a free lunch, it's a different bill: cheap per-token compute traded for a memory floor set by every expert you could route to, not the ones you do. Before picking it over dense, weigh expected sustained volume against that memory floor and the per-expert batch size the model demands. Clear the bar and MoE is close to free money; miss it and you're paying full price for parameters you'll rarely activate.
The correctness question is separate: does the context each request carries reflect the current state of the world? That's what the Inherent Public API is built for: get started in the docs. Sized an MoE deployment and the math surprised you? DM Flow on X.
Next read: Continuous Batching Is How One GPU Serves a Crowd, what MoE does to the batch-size math this post assumes.
