Batch Size Optimizer

The batch size optimizer estimates the largest training batch size that will fit inside a single GPU's memory for a given large language model. Choosing the right batch size is one of the most consequential decisions in deep learning: too small and you waste expensive GPU cycles and get noisy gradients, too large and your job crashes with the dreaded CUDA out-of-memory error. This calculator removes the guesswork by modeling how model weights, activations, and the attention key/value cache compete for the same pool of VRAM.

You provide five inputs that the optimizer reads directly: total GPU memory in gigabytes, model size in billions of parameters, sequence length in tokens, training precision (FP32, FP16, BF16, or INT8), and the number of gradient accumulation steps. From these it derives the model weight footprint, the memory left over for batching, the per-sample memory cost, and finally the maximum batch size the hardware can sustain.

The optimizer first computes how much memory the weights themselves occupy. Each parameter consumes a fixed number of bytes depending on precision — 4 bytes for FP32, 2 bytes for FP16 and BF16, and 1 byte for INT8. Multiplying the parameter count by bytes-per-parameter and dividing by 1024³ gives the model memory in gigabytes. The calculator then reserves 10% of total VRAM as headroom for fragmentation, the CUDA context, and temporary buffers, leaving 90% usable. Subtracting the model footprint from that usable pool yields the available memory for batching.

Per-sample memory is the part most engineers underestimate. As you grow batch size, every additional sample needs its own activation tensors and its own slice of the KV cache. The optimizer estimates a hidden size from the parameter count, then sizes both activation memory and KV-cache memory per sample. Dividing the available batching memory by this per-sample cost gives the maximum batch size, floored to a whole number and clamped to at least 1. Finally, multiplying by your gradient accumulation steps produces the effective batch size — the statistical batch size your optimizer actually sees after accumulating gradients across several micro-batches before each weight update.

Because the tool reports model memory, available memory, memory per sample, tokens per batch, and projected memory utilization side by side, it doubles as a quick memory budget. If utilization sits near 89% you are using the hardware efficiently; if it is far lower, you have room to raise sequence length or batch size, and if available memory goes negative the model simply does not fit in that precision on that card.

The core idea is a memory budget: usable VRAM = model weights + (batch size × memory per sample). The optimizer rearranges this to solve for the largest batch size that does not overflow the card. Understanding each term helps you reason about why a tweak in precision or sequence length moves the answer so much.

Model weights. Weight memory scales linearly with both parameter count and bytes per parameter. Switching a 7-billion-parameter model from FP16 (2 bytes) to INT8 (1 byte) halves the weight footprint, instantly freeing several gigabytes for larger batches. This is why quantization is such a popular lever when memory is tight.

Hidden size estimate. Rather than asking you for architectural details, the optimizer approximates the model's hidden dimension as the square root of the parameter count in billions, multiplied by 1000. A 7B model therefore implies a hidden size near 2,646, while a 13B model implies roughly 3,606. This heuristic captures the empirical relationship between scale and width well enough for planning.

Per-sample memory. Two contributors dominate. Activation memory per sample is modeled as sequence length × hidden size × 4 × bytes-per-parameter, divided by 1024³. The KV cache, which stores attention keys and values across an assumed 32 layers, is modeled as 2 × 32 × hidden size × sequence length × bytes-per-parameter, divided by 1024³. Their sum is the memory each additional sample in the batch demands. Notice that both terms grow with sequence length, so doubling context length roughly doubles per-sample cost and halves the achievable batch size.

Solving for batch size. The optimizer divides available batching memory by per-sample memory, applies a floor to get a whole number of samples, and never returns less than one. It then suggests power-of-two batch sizes (1, 2, 4, 8, 16, 32, 64) that fall at or below the maximum, because powers of two align cleanly with GPU memory layout and tensor-core dimensions, improving throughput.

Getting accurate results from the batch size optimizer depends on entering realistic values. Here is what each field means and how it shifts the outcome.

Precision deserves special attention. FP16 and BF16 both use 2 bytes, so they produce identical memory estimates here, but BF16 offers a wider dynamic range that is more numerically stable for training large models, while FP16 needs loss scaling to avoid underflow. INT8 is typically reserved for inference or quantization-aware setups rather than full backpropagation, but the optimizer still models its single-byte footprint so you can compare scenarios.

The gradient accumulation field is the cleanest way to reach a large effective batch size on memory-constrained hardware. If your card only fits a micro-batch of 12 but you want the smoother gradients of a batch of 48, set accumulation to 4. The GPU processes four micro-batches sequentially, sums their gradients, and applies a single update — statistically equivalent to a batch of 48 while consuming the memory of just 12.

The maximum the calculator returns is a ceiling, not always the ideal choice. Several practical considerations shape the final number you train with.

Leave a safety margin. The optimizer already reserves 10% of VRAM, but real training adds optimizer states (Adam keeps two extra tensors per parameter), temporary kernel workspaces, and memory fragmentation that the simplified model omits. Many teams target a reported memory utilization around 85–90% and then drop one power-of-two step if they hit out-of-memory errors mid-run.

Prefer powers of two. The recommended sizes (1, 2, 4, 8, 16, 32, 64) are not arbitrary. GPU tensor cores and memory controllers are tuned for dimensions that are multiples of 8 or 16, so power-of-two batches often run measurably faster than odd numbers between them, even if both technically fit.

Tune the learning rate with the batch. Larger batches produce lower-variance gradient estimates, which usually means you can — and should — raise the learning rate. A common rule of thumb is the linear scaling rule: when you multiply batch size by k, multiply the learning rate by roughly k, with a short warmup to stabilize the early steps.

Use gradient accumulation for large effective batches. If research suggests an effective batch of 256 but your card only fits 16, set accumulation to 16. The calculator's effective batch size field shows exactly what your optimizer will see, letting you match published recipes without buying more GPUs.

Watch the throughput trade-off. Bigger batches improve hardware utilization up to a point, after which memory bandwidth or kernel overhead dominates and throughput plateaus. The sweet spot is usually the largest power-of-two batch that fits comfortably with headroom, not the absolute maximum. Run a short profiling pass at two or three candidate sizes and pick the one with the best tokens-per-second.

This batch size optimizer is a planning tool that trades precision for speed and simplicity. Knowing its assumptions keeps your expectations realistic.

• Hidden size is estimated. The square-root heuristic (√(params in billions) × 1000) approximates width but will not match every architecture exactly. Mixture-of-experts models, unusually deep or wide designs, and multi-query attention layouts can diverge significantly.
• Layer count is fixed at 32. The KV-cache term assumes 32 transformer layers. Models with far more or fewer layers will have different cache footprints than the estimate suggests.
• Optimizer states are not added. Full fine-tuning with Adam roughly triples the per-parameter memory relative to weights alone. For full-parameter training, treat the result as optimistic and reduce accordingly, or favor memory-efficient methods like LoRA.
• Single-GPU scope. The model assumes one device. Tensor, pipeline, or data parallelism across multiple GPUs changes the math entirely and can support much larger global batches.
• No memory-saving tricks. Activation checkpointing, FlashAttention, paged KV caches, and CPU offloading all reduce real memory use below this estimate, so your actual achievable batch may be higher.

Use the optimizer to get within striking distance of a workable configuration, then confirm empirically with a short test run. If a job crashes with out-of-memory, step down one power of two; if utilization is comfortably below 80%, try stepping up. Treat the reported numbers as a well-reasoned starting estimate that meaningfully narrows the search space rather than an exact memory ledger for your specific framework and kernels.