KV Cache Calculator

The KV Cache Calculator estimates how much GPU memory the key-value cache consumes during autoregressive transformer inference. When a large language model generates text one token at a time, it must remember the key and value projections of every previous token so that the attention mechanism does not recompute them on each step. Those cached tensors are the KV cache, and on long-context workloads they frequently dominate the memory budget more than the model weights themselves.

This calculator takes the architectural parameters that actually drive cache size: the number of transformer layers, the hidden size, the number of query heads, the number of key-value heads (which captures Grouped-Query Attention and Multi-Query Attention), the sequence length, the batch size, and the numeric precision. From these it computes the cache footprint per token, the total cache for the full sequence, the growth rate as generation continues, the memory saved by using fewer KV heads, the bandwidth required to stream the cache, and an estimated maximum batch size for a 24 GB GPU.

Understanding KV cache memory is essential for anyone deploying LLMs in production. The cache grows linearly with both context length and batch size, so a configuration that fits comfortably at 2,048 tokens can overflow a GPU at 32,768 tokens. Engineers serving chat assistants, retrieval-augmented generation pipelines, and high-throughput inference endpoints use these numbers to choose the right GPU, set safe concurrency limits, and decide whether techniques like GQA, MQA, or KV-cache quantization are worth adopting. The calculator turns abstract attention math into concrete gigabytes so you can size hardware before you rent it.

Because the cache is read on every single decoding step, its size also affects latency. Each new token requires the model to read back the entire accumulated cache from high-bandwidth memory, so a larger cache means more bytes moved per token and a higher memory-bandwidth bill. That is why the calculator also reports a bandwidth figure: KV cache is not only a capacity problem but a throughput problem, and both matter when you are optimizing tokens-per-second on real serving hardware.

The core formula multiplies a per-layer, per-token cost across every layer and every token in the sequence and batch. The factor of two accounts for storing both the key tensor and the value tensor. The head dimension is derived from the hidden size divided by the number of query heads, and the number of KV heads (not query heads) sets the cache width, which is exactly how Grouped-Query Attention saves memory.

Per token, the cache for one layer is 2 × KV heads × head dimension × bytes. Multiply by the layer count to get the per-token cache for the whole model, then by sequence length and batch size for the full footprint. With FP16 each element is 2 bytes; INT8 and FP8 cut that to 1 byte, halving the cache.

Multi-Head Attention (MHA) gives every query head its own key and value heads, so the KV cache scales with the full head count. Grouped-Query Attention (GQA) shares each KV head across a group of query heads, and Multi-Query Attention (MQA) takes this to the extreme with a single KV head for all query heads. Because the cache width depends on KV heads rather than query heads, shrinking the KV head count shrinks the cache proportionally without changing the number of query heads doing the attention work.

The calculator reports the memory saved relative to an MHA baseline that uses all query heads as KV heads. The savings percentage follows a simple ratio: with h query heads and kvH KV heads, the cache is reduced to kvH/h of the MHA size, so the saving is 1 − (kvH / h). LLaMA-2-70B, for example, uses 64 query heads with only 8 KV heads, an 8:1 ratio that cuts the KV cache by 87.5% compared to full MHA.

Lower KV head counts trade a small amount of model quality for large memory and bandwidth wins, which is why nearly every modern open-weight model ships with GQA. The calculator lets you set query heads and KV heads independently so you can see the exact saving for your configuration before committing to it.

Numeric precision is the second big lever on KV cache size. FP32 stores 4 bytes per element, FP16 and BF16 store 2 bytes, and INT8 or FP8 store just 1 byte. Switching from FP16 to an 8-bit format halves the cache outright, which is why KV-cache quantization has become a popular way to extend context length on memory-constrained GPUs. The calculator applies the bytes-per-element factor directly, so you can compare formats instantly.

Capacity is only half the story. During generation the model reads back the entire cache from GPU memory on every decoding step, so the cache size sets a hard floor on memory traffic. The calculator estimates the bandwidth required to stream the full cache within a 1-millisecond-per-token target, computed as total cache bytes ÷ 0.001 seconds, then converted to gigabytes per second. If the required bandwidth exceeds your GPU's peak HBM bandwidth, decoding becomes memory-bound and your tokens-per-second will plateau no matter how fast the compute cores are.

This bandwidth-bound behavior is the core reason long-context inference feels slow even on powerful accelerators. Reducing KV heads with GQA and shrinking elements with quantization attack both problems at once: a smaller cache occupies less memory and moves fewer bytes per token, improving capacity and latency together. When you plan a deployment, read the bandwidth figure alongside the total memory figure rather than in isolation.

Enter the architecture of the model you plan to serve. Start with the number of layers and hidden size from the model card, then add the query heads and KV heads; if the model uses standard MHA, set these equal, and if it uses GQA or MQA, set KV heads to the smaller published value. Next set your target sequence length (the full context you intend to support) and the batch size for concurrent requests. Finally pick the precision you will store the cache in.

The result card shows the total KV cache memory in gigabytes or megabytes, along with the cache per token in kilobytes. Below it you will find the head dimension, the GQA memory-saved percentage, the per-token growth rate, the query-to-KV head ratio, and an estimated maximum batch size on a 24 GB GPU after reserving room for FP16 model weights. The memory-by-sequence-length table shows how the footprint scales from 512 up to 32,768 tokens so you can spot exactly where a configuration stops fitting.

1. Look up your model's layer count, hidden size, and head counts.
2. Enter your real serving context length and concurrency in the batch field.
3. Choose FP16 for a typical baseline, or INT8/FP8 to model quantized caching.
4. Read the total memory and bandwidth figures to size your GPU.
5. Lower the KV head count to see the savings from GQA or MQA.

Use the per-token growth number to reason about streaming workloads: every token you generate adds that fixed amount to the cache, so you can predict when a long conversation will exhaust available memory and plan eviction or paged-attention strategies accordingly.

A healthy deployment leaves comfortable headroom between the total KV cache plus model weights and your GPU's physical memory. If the estimated maximum batch size drops to 1, you are already memory-bound and should either reduce the context length, adopt GQA or KV-cache quantization, or move to a larger GPU. The memory saved percentage tells you how much your chosen KV head count is already helping relative to full MHA.

Watch the per-token and growth figures when serving chat or agentic workloads, where context accumulates over many turns. Because the cache grows linearly with sequence length, doubling the context doubles the cache, and the memory-by-sequence-length table makes that scaling obvious. If the bandwidth-required figure is large relative to your GPU, expect decoding to be limited by memory traffic, and treat quantization and GQA as latency tools, not just capacity tools. Together these readouts let you right-size hardware and concurrency for any transformer inference setup.