Model Distillation Calculator

The Model Distillation Calculator estimates the key engineering numbers behind knowledge distillation, the technique that trains a small "student" network to imitate a much larger "teacher" model. Instead of guessing whether a 7B student can usefully replace a 70B teacher, this calculator turns your teacher size, student size, layer counts, dataset size, distillation type, and GPU choice into concrete answers: the compression ratio, the parameter reduction, the GPU memory footprint, the number of accelerators required, the estimated training hours, and the dollar cost of the run.

Knowledge distillation matters because frontier-scale language models are expensive to serve. A teacher with tens of billions of parameters delivers excellent quality but is slow and memory-hungry at inference time. By distilling its behaviour into a compact student, teams keep most of the accuracy while shrinking the serving footprint, cutting latency, and lowering per-request cost. The model distillation calculator quantifies that trade-off up front so you can size hardware and budget before launching a single training job.

The tool models three distillation strategies that practitioners actually use. Logit distillation trains the student to match the teacher's softened output probabilities, the classic "soft label" approach from Hinton's 2015 paper. Feature distillation aligns intermediate hidden states between teacher and student layers, which usually needs more memory and compute. Attention transfer copies the teacher's attention maps and carries the highest overhead of the three. Each choice changes the compute-overhead multiplier and the extra memory the calculator reports, so switching distillation type meaningfully shifts the estimated time and cost.

Every output is derived from the same formulas the calculator runs in JavaScript, so the numbers on the page and the numbers in this guide agree. Memory is computed for FP16 weights, training memory accounts for gradients and optimizer state, and the cost figure combines training hours, the GPUs needed, and a per-GPU hourly price. Use this knowledge distillation calculator as a planning estimate, then validate against a short pilot run on real data before committing to a full schedule.

The headline metric is the compression ratio, computed simply as the teacher parameter count divided by the student parameter count. A 70B teacher distilled into a 7B student gives a compression ratio of 10x, meaning the student is one tenth the size. Because the calculator treats inference speedup as proportional to compression, that same 10x also appears as the estimated inference speedup the smaller model should deliver in production.

Alongside compression, the tool reports parameter reduction as a percentage: the share of the teacher's weights you have eliminated. For the 70B to 7B case the reduction is 90%, and the same value drives the latency-reduction estimate. Larger reductions free up more memory and serving capacity, but they also make the student's job harder, which is where quality retention enters.

The calculator estimates quality retention with a deliberately conservative rule: it starts at 100% and subtracts five percentage points for every unit of compression beyond 1x, then clamps the result so it never drops below a sensible floor or exceeds 100%. A 3.5x compression therefore predicts roughly 87.5% retention, while a 10x compression predicts about 55%. These are intentionally pessimistic planning numbers; well-tuned distillation with good data and temperature often beats them in practice. Treat retention as a "minimum to expect," not a guarantee, and confirm with held-out evaluation.

The practical lesson is that compression and quality pull in opposite directions. Aggressive ratios above 10x maximise savings but risk noticeable accuracy loss on hard tasks, whereas modest ratios near 2x to 4x usually preserve most capability. The model distillation calculator lets you sweep student sizes quickly so you can find the sweet spot where the speedup is worth the quality you give up.

Distillation usually keeps the frozen teacher resident in GPU memory while the student trains, so both models must fit on the hardware at once. The calculator estimates each model's FP16 weight memory as parameters multiplied by two bytes, then divided down to gibibytes. A 70B teacher therefore needs about 130.4 GB just for weights, and a 7B student needs about 13.0 GB.

Training the student costs far more than its raw weights. The tool multiplies the student weight memory by four to approximate the combined cost of weights, gradients, and optimizer state such as Adam's first and second moments. For the 7B student that is roughly 52.2 GB of student training memory. Adding the frozen teacher gives a total memory of about 182.5 GB for the default 70B to 7B run, which is why a single 80 GB card is not enough.

To translate memory into hardware, the calculator divides total memory by ninety percent of each GPU's capacity, reserving a buffer for activations and fragmentation, and rounds up. With 80 GB accelerators the 182.5 GB workload needs three GPUs. The built-in GPU profiles are the NVIDIA H100 80GB, A100 80GB, A10G 24GB, and RTX 4090 24GB, each with its own memory, throughput, and hourly price, so switching cards changes both the GPU count and the cost.

If you cannot afford the GPUs the calculator recommends, the realistic levers are a smaller teacher, a smaller student, offloading the teacher to CPU or precomputing its logits to disk, or activation checkpointing. The knowledge distillation calculator reflects the simplest co-resident setup; advanced memory-saving tricks can reduce the footprint below what it reports.

The calculator estimates compute using the standard transformer rule of thumb that a forward-and-backward pass costs about six floating-point operations per parameter per token. It multiplies six by the student parameters and by a fixed 2048-token sequence length to get FLOPs per sample, then multiplies by the number of training samples and a compute-overhead multiplier that depends on the distillation type: 1.1 for logit, 1.3 for feature, and 1.5 for attention transfer.

To convert raw FLOPs into wall-clock hours, the tool divides total FLOPs by the GPU's peak throughput, by the seconds in an hour, and by a 0.4 utilization factor that reflects realistic 40% efficiency rather than theoretical peak. For the default 70B-to-7B logit run on an A100, that yields roughly 210.6 training hours. Because faster cards have higher throughput, the same workload on an H100 finishes in a fraction of the time.

The cost estimate then multiplies training hours by the number of GPUs needed and by the per-GPU hourly price. The default run spans three A100s at $4.00 per hour, producing about $2,527 for the full job. Note that the cost scales with GPU count, so a memory-driven jump from two to three cards directly raises the bill even if total compute is unchanged.

Two takeaways follow. First, the student size dominates training cost, because FLOPs depend on student parameters, not the teacher; distilling into a tiny student is cheap to train even from a huge teacher. Second, distillation type matters: choosing attention transfer over logit distillation raises the overhead multiplier from 1.1 to 1.5, increasing both hours and dollars by roughly a third for the same data. The model distillation calculator makes these trade-offs visible before you commit budget.

Two distillation hyperparameters that the calculator surfaces are temperature and the layer mapping ratio. Temperature controls how much the teacher's softmax output is softened before the student learns from it. At temperature 1 the probabilities are sharp and the student mostly sees the top prediction; at higher temperatures the distribution flattens, exposing the relative ranking of all classes, the so-called "dark knowledge" that carries the teacher's nuanced understanding.

For most workloads a temperature between 2 and 4 strikes a good balance: high enough to transfer rich inter-class structure, low enough that the signal does not wash out into near-uniform noise. The calculator defaults to a temperature of 2.0, and the soft-label loss is typically scaled by the square of the temperature so its gradient magnitude stays comparable to the hard-label loss. Tuning temperature is one of the cheapest ways to recover quality the conservative retention estimate assumes you will lose.

The layer mapping ratio equals teacher layers divided by student layers and matters most for feature distillation, where intermediate hidden states must be aligned across networks of different depth. With an 80-layer teacher and a 32-layer student the ratio is 2.5, meaning each student layer is mapped to roughly every 2.5th teacher layer. The calculator shows the first few explicit mappings so you can see how student layer one aligns near teacher layer three, student layer two near teacher layer five, and so on.

Good layer mapping spreads supervision evenly through the student's depth rather than clustering it. Uniform spacing, as this tool assumes, is a sound default, but practitioners sometimes weight later layers more heavily because they carry higher-level semantics. The model distillation calculator gives you the ratio and a sample mapping so you can plan feature-alignment loss terms before writing training code.

Start by choosing a distillation type that matches your goal. Pick logit distillation for the simplest, cheapest setup that works well for most language tasks; choose feature distillation when you need the student to internalise intermediate representations; choose attention transfer when reproducing the teacher's attention patterns is critical. Your choice sets the compute-overhead multiplier and the extra memory the page reports.

Next, enter your teacher and student sizes in billions of parameters along with their layer counts. The calculator immediately shows the compression ratio, parameter reduction, and estimated quality retention, letting you judge whether the student is large enough for your accuracy target. If retention looks too low, increase the student size and watch the trade-off shift.

Then set the training samples and pick a GPU type. The dataset size drives total FLOPs and therefore training hours and cost, while the GPU choice changes throughput, memory, hourly price, and the number of cards required. Finally, adjust the temperature if you plan to deviate from the standard 2 to 4 range. Read off the memory breakdown, GPUs needed, training time, and estimated cost, and iterate on inputs until the plan fits your hardware and budget. Use this knowledge distillation calculator for fast scenario planning, then confirm the critical numbers with a short pilot run on representative data.