Speculative Decoding Calculator

The Speculative Decoding Calculator estimates how much faster a large language model can generate text when a small, cheap draft model proposes several tokens at once and a large, accurate target model verifies them in a single forward pass. Speculative decoding is one of the most effective lossless inference acceleration techniques for modern LLMs: it produces exactly the same output distribution as standard autoregressive decoding, but trades a few inexpensive draft passes for fewer expensive target passes. This tool turns that trade-off into concrete numbers so you can decide whether the technique is worth deploying for your workload.

To use the calculator you provide the target model size in billions of parameters, the draft model size, the speculation length K (how many tokens the draft proposes per round), the acceptance rate alpha (the probability that a proposed token is accepted by the target), the target latency and draft latency in milliseconds per token, and the expected output length in tokens. From these seven inputs it computes the expected accepted tokens per round, the number of verification rounds needed, the total speculative decoding time versus standard decoding time, the resulting inference speedup, throughput in tokens per second, the draft model utilization, the memory overhead of hosting the draft alongside the target, and a sweep showing how speedup changes as the acceptance rate climbs from 50% to 90%.

Engineers serving chat assistants, code completion endpoints, and high-throughput batch pipelines use a speculative decoding calculator to answer practical questions: Is my draft model fast enough relative to the target? Will a longer speculation length actually help, or will rejections waste the extra draft passes? How much extra GPU memory does the draft model cost? By making the speedup math explicit, the calculator helps you choose a draft-target pairing, tune the speculation length K, and set realistic latency expectations before you commit hardware to the deployment.

Because every accepted token skips a full target forward pass, the headline metric is the inference speedup: the ratio of standard autoregressive time to speculative time. A speedup of 1.8x means the same response arrives in roughly 55% of the original wall-clock time. The calculator also reports the target call reduction, which shows how many of the original per-token target passes you avoid, the single biggest driver of cost savings on memory-bandwidth-bound inference hardware.

Standard autoregressive generation produces one token per target forward pass, so a 256-token response requires 256 sequential passes through the large model. Speculative decoding breaks this serial bottleneck. In each round, the small draft model autoregressively generates K candidate tokens. The target model then runs one parallel forward pass that scores all K candidates simultaneously. Using a rejection-sampling rule, the target accepts the longest matching prefix of candidates and corrects the first mismatch, guaranteeing the output matches what the target alone would have produced.

The efficiency hinges on the acceptance rate alpha. If the draft agrees with the target on most tokens, long prefixes get accepted and many target passes are skipped. The expected number of accepted tokens follows a geometric series: each successive candidate is accepted only if all earlier ones were, so the contribution of position i is alpha raised to the power i. Summing over K positions gives the expected accepted count. The calculator adds the bonus token the target itself emits on each verification, captures the cost of K draft passes plus one target pass per round, and divides the output length by the tokens produced per round to find how many rounds the full generation needs.

This is why the draft model must be both fast and well-aligned. A draft that is too slow erodes the savings from skipped target passes, while a draft that disagrees too often wastes its proposals. The sweet spot is a draft 5-15x smaller than the target that shares the same tokenizer and training distribution, typically yielding acceptance rates of 0.6-0.85 in practice.

The calculator builds the speedup from a chain of formulas that exactly mirror the math used in autoregressive verification. First it computes the expected accepted tokens per round as a truncated geometric series, then the time per round (K draft passes plus one target pass), then the tokens produced per round, the rounds needed, and finally the speedup as the ratio of standard time to speculative time.

Standard decoding time is simply the output length times the target latency. Speculative time is the number of rounds times the per-round cost. Each round produces the expected accepted tokens plus a correction token the target supplies whenever the round did not accept all K candidates, an event with probability 1 − alpha^K.

The two levers you control most directly are the acceptance rate alpha and the speculation length K. Acceptance rate measures how often the draft model's guess survives the target's verification. A higher alpha means longer accepted prefixes, fewer rounds, and a larger speedup. Acceptance is driven by how closely the draft mimics the target: same tokenizer, same domain, and a draft large enough to capture the easy, predictable tokens that make up much of natural text.

Speculation length K sets how aggressively you gamble. Increasing K can accept more tokens per round when alpha is high, but it also adds draft latency to every round and yields diminishing returns because the probability of accepting all K candidates, alpha^K, decays geometrically. With alpha = 0.7, the chance of accepting four straight tokens is only 0.7⁴ = 0.24, so pushing K to 8 or 10 rarely pays off. The calculator includes an optimal K hint that balances the geometric acceptance decay against your draft-to-target latency ratio.

The built-in acceptance-rate sweep makes the relationship visible. Holding K, latencies, and output length fixed, raising alpha from 0.5 to 0.9 dramatically increases tokens accepted per round and therefore speedup. This is why teams invest in aligning the draft model through distillation or shared-vocabulary training: a few points of acceptance rate translate into meaningful latency reductions across millions of requests.

Speculative decoding is not free in memory. You must host the draft model in GPU memory alongside the target. The calculator estimates each model's footprint assuming FP16 weights at 2 bytes per parameter, so a 70-billion-parameter target needs about 140 GB and a 7-billion-parameter draft about 14 GB. The memory overhead is reported as the draft footprint expressed as a percentage of the target footprint, which equals the draft-to-target parameter ratio. A 7B draft paired with a 70B target adds roughly 10% memory overhead.

The calculator also reports draft utilization, the expected accepted tokens divided by K. This tells you what fraction of the draft's proposed tokens actually survive verification. Low utilization means the draft is generating tokens that get thrown away, wasting compute. High utilization means the draft is well-matched and most of its work contributes to the final output. Combined with the target call reduction metric, which reports the percentage of original per-token target passes you avoid, these efficiency numbers help you judge whether the memory cost of the draft is justified by the latency win.

For memory-constrained deployments, the draft size is the key knob. A smaller draft adds less memory and lower per-round latency, but may align less well and lower the acceptance rate. The calculator lets you sweep both dimensions to find a draft that fits your GPU budget while still delivering a worthwhile speedup.

Speculative decoding shines in latency-sensitive, single-stream serving where a user waits for a response token by token, such as interactive chat, coding copilots, and agentic tool-use loops. In these settings the target model is memory-bandwidth bound, meaning the GPU spends most of each step reading weights rather than doing math, so verifying K candidates in one pass costs barely more than verifying one. That free parallelism is exactly what speculative decoding exploits.

The technique is less compelling when the target is already compute-bound, for example at very large batch sizes where the GPU is saturated and the extra verification work competes for the same arithmetic units. It also underperforms on highly creative or out-of-distribution generation where the draft's acceptance rate collapses. Before deploying, use this speculative decoding calculator to confirm that your draft latency, acceptance rate, and output length combine into a speedup above 1.0; if the numbers say 1.1x, the added memory and operational complexity may not be worth it, but a clean 1.8x-2.5x on a chat workload usually is.

Modern frameworks such as vLLM, TensorRT-LLM, and Hugging Face Transformers ship speculative decoding (sometimes called assisted generation) out of the box, and self-speculative variants like Medusa and EAGLE remove the need for a separate draft model entirely. Whichever path you choose, the speedup math in this calculator gives you a quick, vendor-neutral estimate of the payoff.