Inference Latency Calculator

The inference latency calculator estimates how long a large language model (LLM) takes to respond to a single request, broken into the two phases that every transformer goes through: prefill (reading your prompt) and decode (generating the answer one token at a time). Latency is the single most important user-facing metric for chatbots, copilots, RAG pipelines, and any real-time AI feature, because it determines whether a response feels instant or sluggish.

This LLM latency calculator takes six inputs that map directly to a real deployment: the model size in billions of parameters, the number of input tokens (your prompt), the number of output tokens (the response you expect), the GPU type running the model, the quantization precision, and the batch size of concurrent requests. From these it derives a sustained generation speed in tokens per second, then converts that into wall-clock milliseconds.

Four numbers come out the other side. Time to first token (TTFT) tells you how long the user waits before any text appears. Inter-token latency (ITL) is the gap between successive streamed tokens and governs how smoothly text flows. Total response latency is the full end-to-end time, and effective throughput is the average number of tokens processed per second across the whole request. Together these tell you whether a configuration will feel snappy or frustrating before you ever provision a GPU.

Because the calculator separates prefill from decode, it also explains why latency behaves the way it does. Long prompts inflate TTFT but barely touch inter-token latency, while long responses dominate total time because every output token is generated sequentially. Understanding that split is the key to optimizing real inference workloads, and it is exactly what this tool is built to surface.

The model starts from a published baseline of tokens per second for a 7B model at FP16 on each supported GPU, then scales that baseline by four multipliers. A quantization multiplier rewards lower precision (INT4 runs faster than FP16), a size multiplier penalizes larger models using a square-root relationship, and a batch-efficiency factor models the throughput gain from processing multiple requests together, also as a square root.

Once the sustained generation speed is known, the calculator splits the request into prefill and decode. Prefill processes the entire prompt in parallel and is assumed to run roughly 5x faster than autoregressive generation, so the prompt is divided by five times the base speed to get time to first token. Decode then divides the output tokens by the plain generation speed, because each output token depends on the previous one and cannot be parallelized.

Inter-token latency is simply the reciprocal of the generation speed expressed in milliseconds, and effective throughput averages all tokens (input plus output) over the total wall-clock time. Every result you see on the page comes from these closed-form expressions, so the math is fully transparent and reproducible.

The quantization multipliers are FP32 = 0.5, FP16 = 1.0, BF16 = 1.0, INT8 = 1.5, and INT4 = 2.5, reflecting that lower-precision arithmetic moves less data through memory and runs faster on tensor cores.

Every LLM request has two distinct phases, and the inference latency calculator models both separately because they scale in completely different ways. The prefill phase ingests your full prompt at once. Because the transformer can attend to all prompt tokens in parallel, prefill is compute-bound and finishes quickly, which is why the calculator treats it as roughly five times faster than generation. Prefill time is exactly what the user experiences as time to first token.

The decode phase is fundamentally serial. Each new output token is produced from the model's prediction for the previous token, so the GPU cannot generate token 200 until it has generated token 199. This autoregressive bottleneck makes decode memory-bandwidth-bound and is the reason long responses dominate total latency. The gap between two streamed tokens is the inter-token latency, and at 100 tokens per second that gap is a steady 10 milliseconds.

This split has practical consequences. If your prompt is huge but the answer is short (for example a long RAG context summarized into one sentence), TTFT will be large but total latency stays modest. If your prompt is tiny but the answer is long (such as generating a full article), TTFT is near-instant yet total latency balloons because hundreds of tokens are decoded one after another. Knowing which phase dominates tells you whether to optimize context length, switch to a faster GPU, or apply quantization to speed up decoding.

Batching and speculative decoding both target this serial bottleneck. The batch-size input in the calculator raises throughput via a square-root efficiency curve, modeling how a GPU amortizes weight loads across several requests, even though each individual request still pays its own decode cost. Understanding prefill versus decode is the foundation for every latency optimization you will make in production.

Six levers control inference latency, and this calculator lets you adjust all of them. Model size matters most: doubling parameters slows generation by the square root of the ratio, so a 28B model runs at half the speed of a 7B model on the same hardware. Larger models read more weights from GPU memory for every single token, and that memory traffic is the dominant cost during autoregressive decoding.

GPU type sets the baseline. An H100 generates a 7B model at about 150 tokens per second while a T4 manages only 25, a six-fold difference that flows straight through to TTFT, inter-token latency, and total response time. Quantization trades numerical precision for speed: moving from FP16 to INT8 multiplies throughput by 1.5x, and INT4 multiplies it by 2.5x, because smaller weights move less data through the memory subsystem.

Prompt length drives prefill cost and therefore TTFT, while output length drives decode cost and therefore the bulk of total latency. Finally, batch size improves aggregate throughput, modeled here as a square-root gain, because the GPU can load model weights once and reuse them across several concurrent requests. The table below summarizes how each input pushes latency up or down.

Once the inference latency calculator shows where your time is going, you can attack the largest component first. If total latency is dominated by generation, the highest-leverage move is usually quantization: switching from FP16 to INT8 or INT4 multiplies generation speed by 1.5x or 2.5x in this model, cutting both inter-token latency and total response time with minimal quality loss for many tasks.

If TTFT is your problem, shorten the prompt. Trimming retrieved context in a RAG system, compressing system instructions, or caching repeated prefixes all reduce the number of input tokens the model must read before producing the first word. Because prefill runs in parallel, even large reductions here translate into a visibly snappier first response.

Upgrading the GPU is the bluntest instrument and the most reliable: moving a workload from a T4 to an A100 80GB roughly quadruples base throughput, and an H100 adds another 50% on top. Choosing a smaller model that still meets your quality bar is equally powerful, since the square-root size penalty means a 7B model can be more than twice as fast as a 30B model on identical hardware.

For high-traffic services, increasing batch size raises aggregate throughput so you serve more users per GPU, even though individual request latency is not reduced by batching alone. The right strategy is almost always a combination: pick the smallest acceptable model, quantize it, run it on the fastest GPU your budget allows, keep prompts lean, and batch concurrent traffic. Re-run the calculator after each change to confirm the win before committing hardware.

The calculator returns four headline numbers, and each answers a different operational question. Total response latency is what a user waits for a complete, non-streamed answer; for interactive chat you generally want this under a few seconds. Time to first token matters most for streamed UIs, where the perception of speed depends on how quickly text begins to appear rather than when it finishes.

Inter-token latency determines whether streamed text reads smoothly or stutters; below about 50 milliseconds per token, output appears fluid to most readers, and at 10 milliseconds per token (100 tok/s) it streams faster than anyone can read. Effective throughput blends input and output over the whole request and is the right metric for capacity planning, because it reflects total tokens moved per second rather than just decode speed.

Read the latency breakdown to find your bottleneck. A large prefill time relative to generation means your prompt is the constraint, so context trimming or prefix caching will help most. A large generation time means the response length and decode speed dominate, pointing you toward quantization, a faster GPU, or a smaller model. These estimates are planning figures based on representative tokens-per-second baselines, so treat them as a strong starting point and validate against your own benchmarks before finalizing a production deployment.