Mixture of Experts (MoE): what it is, how it works, and why it matters in 2026

August 11, 202610 minAI, MoE, Mixture of Experts, Qwen, Kimi, Architecture, LLM

Short answer (60 seconds): Mixture of Experts (MoE) is an architecture where each transformer layer has many small modules called experts (typically FFNs) and a router (gating network) that decides which experts to activate for each token. Only the selected experts compute — the rest of the model lives in memory but does not participate in the computation. The key metric is active vs total parameters: Kimi K2 has 1T total but only 32B active per token; Qwen3-30B-A3B has 30.5B total and 3.3B active. Why it matters: it gives you higher total capacity at the same per-token FLOP cost as a much smaller dense model. The real tradeoff: training requires load balancing between experts (auxiliary loss, expert collapse), and inference is more expensive in memory and communication because all experts must live in VRAM even though you only use some.

If you want to understand why the LLM conversation in 2026 increasingly revolves around MoE — and why knowing the difference between "30B parameters" and "3B active" changes how you evaluate a model for your SaaS — this post is for you. I will explain the architecture without assuming you already know what an FFN is, I will verify the Qwen and Kimi numbers (not from memory), and I will be honest about the tradeoffs that papers tend to hide.

Why MoE matters in 2026

The numbers speak for themselves:

What the table tells you is counterintuitive at first: a model with 30x more total parameters can be cheaper to run per token than one with far fewer. It is the difference between capacity (how much information the model can store) and compute per token (how much each inference costs you). MoE separates the two; dense models cannot.

The original MoE paper is from 1991 (Jacobs et al.), but it was not until 2017 that Shazeer et al. applied it to a 137B-parameter LSTM in production. The 2024-2026 difference is that MoE is now the default pattern for frontier open-weight models (Qwen, DeepSeek, Kimi, Mixtral) and likely for several proprietary ones (rumored GPT-4, speculated Claude).

What MoE is exactly

Intuition first: imagine a general hospital with a single doctor who has to attend everything. Now imagine a hospital with 128 specialist doctors and a receptionist who, based on symptoms, routes the patient to the right specialist. The second hospital sees the same number of patients per hour (each patient only sees one specialist) but has more installed capacity for rare cases because there is a cardiologist, a dermatologist, a neurologist, etc.

MoE is the neural equivalent of the second hospital:

• Experts: instead of a dense FFN per transformer layer, you have N experts (typically N between 8 and 384). Each expert is a small FFN — roughly the size of a dense FFN divided by N.
• Router (or gating network): a small linear layer (a single nn.Linear) that takes the token's hidden state and produces N logits. From those logits, the router selects the top-k experts for that token.
• Forward pass: only the selected k experts compute their output. Their outputs are combined (typically a softmax-weighted sum) and passed to the next layer.

For a 70B dense model, every token passes 70B weights through computation. For a 1T-total MoE with 32B active, each token only passes 32B — but the model has 1T of "memory" to store knowledge.

The architecture visualized

The critical thing in the diagram is what does not appear: the non-selected experts compute nothing. In Qwen3-30B-A3B, out of 128 experts only 8 do work per token. The other 120 sit idle.

Total vs active parameters: the metric that changes everything

This is the most important distinction in the post. If your provider or paper says "30B parameters", it does not tell you how much it costs you. You need both numbers:

• Total parameters: everything stored. Defines the model file size, the minimum VRAM to load it, and — in practical terms — how much knowledge it can potentially encode.
• Active parameters: the fraction that participates in a token's forward pass. Defines the FLOPs per token, inference speed, and computational cost per request.

For current MoE models:

Qwen3-30B-A3B (verified from the official HuggingFace model card):

Kimi K2 (verified from the official MoonshotAI GitHub and technical report):

Kimi K2's activation ratio (~3.2%) is extreme and is part of why Moonshot chose the Muon optimizer over standard AdamW: at that scale, optimizer efficiency matters a lot more.

Why does this distinction matter to you? Because when a provider charges you per million tokens or promises you a certain speed, what matters is the FLOPs per token — and that scales with active, not total. A "1T parameter" model can be cheaper per token than a "70B parameter" one if its activation is small.

Why MoE scales better than dense models

The core argument from the Mixtral paper (Mistral AI, Dec 2023) and from several DeepSeek analyses is this: for a fixed training FLOP budget, an MoE with many experts optimized for sparsity learns faster than a dense model with fewer parameters.

The formal intuition: in a dense model, adding parameters increases cost per token linearly. In an MoE, adding experts increases total parameters (more knowledge storage capacity) but does not increase FLOPs per token (because top-k stays constant). So, for the same FLOPs per token, you can have 5-10x more capacity.

The Mixtral paper reported that Mixtral 8x7B outperforms Llama 2 70B on most benchmarks with 6x faster inference. The reason: Mixtral has ~12.9B active vs 70B dense — and FLOPs per token drop proportionally.

But there is an important "but" that many papers minimize: more capacity without more FLOPs is not free in training. The model needs more tokens to fill that capacity, which raises the total training run cost. MoE wins on per-token efficiency but not necessarily on absolute total cost.

Training challenges: why MoE is not trivial

Three concrete problems that any team training MoE has to solve:

1. Load balancing. If you let the router train freely, it collapses: all tokens go to the same expert (typically the one with the highest initial logits) and the other experts starve. The standard solution is an auxiliary loss that penalizes imbalanced distributions and forces each expert to receive a similar fraction of the traffic. Shazeer et al. introduced the original load balancing loss; Mixtral and Qwen use top-k variants.

2. Expert collapse. Symptom of failed load balancing: during training, some experts stop receiving tokens and their gradients go to zero. They end up representing dead knowledge. Modern variants include noisy routing (add noise to router logits during training) and capacity factors (limit how many tokens an expert can handle per batch) to prevent it.

3. Communication overhead in distributed training. When you train MoE with 8 or more GPUs, dispatch (sending tokens to their assigned experts) and combine (collecting results) requires all-to-all operations that dominate training time at scale. DeepSeek-V3 reported specific optimizations (DualPipe, fine-grained expert parallelism) to mitigate this.

Inference challenges: why MoE is more expensive to serve

If training is hard, inference has its own problems that often only show up in production:

1. Total memory, not active. Even though only 3.3B of Qwen3-30B-A3B's parameters are active per token, all 30.5B must live in VRAM. For Kimi K2 with 1T parameters in FP16, you need 2TB of VRAM distributed across GPUs (typically 16-32 H100/H200 accelerators). The capital cost of serving MoE is high even if per-token compute is low.

2. All-to-all communication. In inference with experts distributed across multiple GPUs, every generated token requires all-to-all communication: the router tells each GPU "your experts attend these tokens" and then collects the results. The latency of that communication is fixed and dominates total time when experts live on several GPUs.

3. Bimodal latency. In production, if some experts become popular for certain query types (e.g., the "code expert" gets much more traffic than the "poetry expert"), the GPUs hosting those experts get congested. Latency becomes bimodal: fast requests and slow requests, with no reliable mean value. Monitoring and mitigating this requires dynamic batching or speculative expert prefetching.

4. KV cache fragmentation. Each generated token requires recomputing the router and maintaining the KV cache. Since routing changes based on context, the memory access pattern is irregular and inference engines (vLLM, SGLang) need MoE-specific optimizations that do not apply to dense models.

For a SaaS founder, the practical translation is: MoE via API is generally a good quality/price ratio, but self-hosting a large MoE is usually unfavorable in TCO until your volume justifies the investment in distributed infrastructure.

When to choose MoE over a dense model

Three practical rules I use with clients:

Pick MoE if:

• Your provider offers it via API (you do not need to self-host) and the quality per dollar is better than dense alternatives. Example: Qwen3-30B-A3B is competitive with 14B-30B dense models on many benchmarks but runs at 3B dense speed.
• You are doing fine-tuning and want more capacity without paying the cost of a larger dense. MoE fine-tuned with LoRA is surprisingly efficient.
• Your workload is agentic or long-horizon coding where the model's "large memory" matters more than minimum latency per token.

Pick dense if:

• You need to self-host and do not want to handle expert parallelism complexity. A 7B-13B dense runs on a single modern GPU; an equivalent MoE almost always requires multiple.
• Your workload is extreme latency-sensitive (e.g., full-duplex voice). The variability of MoE routing is a problem when every millisecond counts.
• You need fine-grained control over routing or detailed architecture-level debugging. Dense is easier to inspect.

Pick a small dense or small MoE (3-8B active) if:

• You are prototyping. Iteration cost is what matters, not peak capacity.

Conclusion: MoE is the 2026 default, but it is not magic

Mixture of Experts is the dominant architecture for frontier models in 2026 and will continue to be. But "MoE" is not a binary property — it is a spectrum of decisions: how many experts, how many active per token, what type of routing, what auxiliary losses, how to distribute across hardware. When a provider tells you "we use MoE", always ask for the activation ratio. That is the number that matters.

If your SaaS already consumes frontier models via API and you still do not understand the difference between total and active parameters, now is the time to learn it. It is the difference between paying USD 0.50 and USD 11 for the same task completed.

Want to discuss how MoE fits your architecture before picking a model for production? There is a CTA at the end with a free 30-minute call.

Frequently asked questions

What is Mixture of Experts (MoE) in one sentence?

It is a neural network architecture where, instead of a single dense FFN per layer, you have many small modules called "experts" and a router that activates only the k most relevant ones for each token. The model has many total parameters, but only a fraction is active in any given inference.

What is the difference between total and active parameters?

Total parameters = everything stored in model memory (all experts + embeddings + attention). Active parameters = the fraction that participates in a token's forward pass (the k experts selected by the router + attention). Kimi K2 has 1T total and 32B active: per token, it only computes 3.2% of its weights.

Does MoE make training cheaper?

Not directly. Training cost scales with active parameters per token (forward + backward FLOPs), not total. So MoE gives you more total capacity at the same per-token training cost, but total training cost goes up because you need more tokens to fill that capacity. The right metric is FLOPs per token, not total parameters.

Why is MoE harder to serve (inference) than a dense model?

Three reasons. (1) Memory: all experts must live in VRAM even if you only use some per token — for 1T parameters in FP16 you need 2TB of distributed VRAM. (2) Communication: the router does dispatch and combine between GPUs (all-to-all), which is pure latency. (3) Load balancing in production: if the token-to-expert distribution skews, popular experts become bottlenecks and latency becomes bimodal.

How many active experts do Qwen3-30B-A3B and Kimi K2 use?

Qwen3-30B-A3B uses 128 total experts and activates 8 per token (top-8, no shared expert). Kimi K2 is more extreme: an MoE scheme with 32B active over 1T total — I don't have the exact expert count for K2 public at the time of this post, but the activation ratio is ~3%.

When should I choose MoE over a dense model?

When your metric is capacity per FLOP (training compute or inference per token) and you can tolerate the fixed memory cost and serving complexity. For self-hosting, MoE is usually unfavorable because you need GPUs with lots of VRAM even though you only activate few experts. For API-based inference (where the provider optimizes serving), MoE gives you more quality per dollar.

Is MoE the same as "sparse models"?

MoE is the most popular implementation of sparse models. Sparse means that, per token, only a fraction of the parameters participates in the computation. MoE achieves this with the router that selects experts; other sparse techniques (like sparse attention) operate on different dimensions of the architecture.