Mixture of Experts Calculator

The Mixture of Experts calculator estimates how big a sparse MoE transformer is, how much GPU memory it needs, and how efficiently it spends compute compared to a dense model of the same quality. Mixture of Experts is the architecture behind sparse large language models such as Mixtral, DeepSeek-V3, and Qwen MoE, and its defining trick is simple: a layer holds many parallel feed-forward "experts," but a lightweight router sends each token to only a small subset of them. That means the model can carry an enormous total parameter count for capacity while keeping the active parameter count per token low for speed.

This MoE calculator turns that idea into concrete numbers. You enter the hidden size, number of transformer layers, total experts, active experts (the top-K routed per token), the expert intermediate (FFN) size, the vocabulary size, an optional shared-expert ratio, and the weight precision. From those inputs it reports total parameters, active parameters per forward pass, full-model memory, active memory, the per-expert and per-layer parameter breakdown, and efficiency metrics such as parameter efficiency, expert utilization, and memory overhead. The tool assumes a SwiGLU-style expert with three projection matrices (gate, up, and down), which is the standard in modern sparse models.

Why does this matter for anyone building or serving language models? Because MoE decouples capacity from compute. A model can advertise 47 billion total parameters yet only fire roughly 13 billion of them for any given token, delivering the modeling quality of a much larger dense network at a fraction of the FLOPs. The catch is memory: every expert weight still has to live in VRAM even when it is not selected, so storage scales with total parameters while compute scales with active parameters. Understanding that split is the whole point of using a mixture of experts calculator before you commit to a hardware budget or a training run.

The calculator builds the parameter total from four ingredients per layer plus a shared embedding table. Getting the breakdown right is what separates a useful mixture of experts calculator from a back-of-the-envelope guess.

Attention. Each layer carries the same self-attention block as a dense transformer: query, key, value, and output projections. Together these are 4 × d² parameters, independent of how many experts you add.

Experts. Every expert is a SwiGLU feed-forward network with a gate matrix, an up matrix, and a down matrix. Each matrix is roughly d × ff, so one expert holds 3 × d × ff parameters. With E experts per layer, the expert block is E × 3 × d × ff parameters — the part that dominates the total in any large sparse model.

Router. A tiny linear gate maps the hidden state to a score for each routed expert, costing d × routedExperts parameters. This is a rounding error compared with the experts themselves, which is exactly why MoE can scale capacity so cheaply per token.

Layer norm and embeddings. Normalization adds 4 × d parameters per layer, and the model includes input and output embeddings of 2 × V × d parameters shared across the network.

The single most important number a mixture of experts calculator reports is the gap between total parameters and active parameters. Total parameters count every weight stored on disk and in memory. Active parameters count only the weights a single token actually touches during a forward pass: the full attention block, the router, layer norms, embeddings, plus the K experts the router selected (and any shared experts that always fire).

Because only K of E experts run per token, the expert compute drops by a factor of K/E. The calculator computes active expert parameters as K × 3 × d × ff (plus shared experts), then adds the always-on components to get active parameters per layer. Multiply by the number of layers and add embeddings, and you have the active total. The parameter efficiency metric is simply active parameters divided by total parameters, expressed as a percentage — a low value means the model is highly sparse and stores far more capacity than it uses per token.

This split drives every practical decision about a sparse model. Inference latency and training FLOPs track active parameters, so an MoE behaves like a small model on the math. But VRAM must hold all experts at once, so memory tracks total parameters. The memory overhead metric captures that tension: it is the extra memory the full model needs relative to the active footprint. A model that is wonderfully cheap to compute can still be expensive to host, which is why teams reach for quantization (INT8 or INT4) to shrink the resident weights. Use the precision selector in the calculator to see how dropping from FP16 to INT4 roughly quarters total memory while leaving the parameter counts unchanged.

Beyond raw parameter counts, the MoE calculator surfaces three efficiency signals that help you reason about a design before you build it.

Expert utilization is K divided by E. With a top-2 router over 8 experts you get 25%, meaning each token engages a quarter of the available expert capacity. Lower utilization gives more sparsity and a larger capacity-to-compute ratio, but routing very few experts can hurt quality and load balancing if the router collapses onto a handful of favorites.

Parameter efficiency tells you what fraction of the stored network does useful work per token. A figure near 30% is typical for an 8-expert top-2 design; it signals that roughly 70% of the weights are dormant on any given step, waiting in memory for tokens routed elsewhere.

Memory overhead expresses how much extra memory the full model demands over the active footprint. The calculator also reports an equivalent dense model: a hypothetical dense network with the same active parameter count, which is a fair proxy for the quality-per-compute you are buying. If the equivalent dense model is, say, 13B while the total is 47B, you are getting 13B-class speed with the headroom of a far larger network.

Finally, the tool estimates tokens per expert using a capacity factor of 1.25, a common default that gives each expert a little slack to absorb uneven routing without dropping tokens. In production, a capacity factor that is too tight drops tokens during bursts, while one that is too loose wastes compute on padding. Reading these numbers together — utilization, efficiency, overhead, and capacity — gives a rounded picture of whether a mixture of experts configuration is balanced or lopsided before you spend a single GPU-hour.

Planning hardware for a sparse model is where the mixture of experts calculator earns its keep. The full-model memory equals total parameters multiplied by bytes per parameter, divided by 1024³ to get gigabytes. The calculator uses 4 bytes for FP32, 2 bytes for FP16 and BF16, 1 byte for INT8, and 0.5 bytes for INT4. Because experts dominate the count, total memory grows almost linearly with the number of experts even though compute does not.

That asymmetry has real consequences. A top-2-of-8 model is cheap to run but still has to fit all eight experts per layer in VRAM, so you may need multiple high-memory GPUs purely for storage. Quantization is the usual lever: moving weights from FP16 to INT4 cuts the resident size by roughly 4× with modern schemes, often turning a multi-GPU deployment into a single-card one. The calculator's active-memory figure also matters for expert-parallel serving, where experts are sharded across devices and only the selected ones are gathered per token — though every device must still hold its slice of every expert.

When you compare an MoE against a dense alternative, weigh three costs separately: storage (scales with total params), compute (scales with active params), and communication (the all-to-all routing traffic that grows with the number of devices). The calculator addresses the first two directly and lets you reason about the third by showing how aggressive your sparsity is. Treat its memory output as the floor for weights alone; real deployments add the KV cache, activations, and optimizer states during training, so budget extra headroom on top of the number shown here.