Model Parameter Count Calculator

The Model Parameter Count Calculator estimates the total number of trainable weights inside a decoder-only transformer language model and the GPU memory those weights require at several numerical precisions. Instead of digging through a model's configuration file or running a Python script, you enter six architecture values: the hidden size (d_model), the number of layers, the number of attention heads, the vocabulary size, the intermediate (feed-forward) size, and the architecture family. The calculator then returns the parameter count in billions, the raw parameter total, and the storage footprint in FP32, FP16, INT8, and INT4.

This is the fastest way to answer the questions that come up constantly in LLM engineering: How big is this model, really? Will it fit on my GPU? How much smaller does quantization make it? A 4096-hidden, 32-layer LLaMA-style configuration resolves to roughly 6.74 billion parameters and about 12.55 GB in half precision, which is exactly why 7B-class models are popular on 16 GB and 24 GB consumer cards. Because the calculator breaks the total into embedding, attention, and feed-forward (FFN) buckets, you can see at a glance where your parameter budget is being spent and which architectural knob to turn when you need a model of a target size.

Whether you are reverse-engineering a published model card, sizing a custom architecture before a training run, or planning inference hardware, this parameter count calculator gives a transparent, formula-driven estimate that matches the standard transformer parameter accounting used across the deep learning community.

The calculator sums parameters from three sources and then adds the embedding table twice, once for the input token embeddings and once for the output (unembedding) projection. Many modern models tie these weights, but the upper-bound accounting here counts them separately, which is how the tool reports its total.

Embedding parameters are simply the vocabulary size multiplied by the hidden size: a lookup table with one d_model-dimensional vector per token. Attention parameters per layer come from four square projection matrices, the query, key, value, and output projections, each of size d × d, giving 4d². Feed-forward (FFN) parameters depend on the architecture: a standard GPT-style block uses an up and a down projection (2 × d × ff), while LLaMA and Mistral use a SwiGLU gate that adds a third matrix (3 × d × ff). Two layer-norm vectors per layer contribute a tiny 2 × 2 × d.

The per-layer subtotal is multiplied by the number of layers and added to both embedding tables to produce the grand total. The tool then converts that count into memory by multiplying by bytes per parameter: 4 for FP32, 2 for FP16, 1 for INT8, and 0.5 for INT4, then dividing by 1024³ to express the result in gibibytes.

One of the most useful outputs of this calculator is the parameter distribution, which separates the total into embedding, attention, and FFN contributions. Understanding this split helps you reason about scaling. As models grow deeper and wider, the embedding table becomes a smaller fraction of the whole, while the feed-forward block dominates because it scales with both d and ff.

For the default 7B-class configuration, the FFN block carries the heaviest load at roughly 135 million parameters per layer versus 67 million for attention, which is why the common rule of thumb sets the intermediate size to about 2.7 times the hidden size in SwiGLU models. The embedding table, at around 131 million parameters per copy, looks large for a single layer but quickly becomes a minority share once you stack 32 transformer blocks.

Parameter count alone does not tell you whether a model fits on your hardware; the numerical precision of the stored weights does. The calculator translates the parameter total into a memory footprint by assuming a fixed number of bytes per weight. FP32 (full precision) uses 4 bytes, FP16 and BF16 (half precision) use 2 bytes, INT8 uses 1 byte, and INT4 uses half a byte. Each step down roughly halves the memory you need to hold the weights.

For a 6.74B-parameter model the footprints work out to about 25.10 GB in FP32, 12.55 GB in FP16, 6.28 GB in INT8, and 3.14 GB in INT4. This is why a 7B model that cannot run in full precision on a 24 GB card slots comfortably into 12-16 GB at half precision and runs on modest laptops once quantized to 4-bit. Keep in mind that these figures cover weights only: actual inference also needs memory for activations and the KV cache, and training adds optimizer states and gradients that can multiply the requirement several times over.

Use the precision table to plan deployments: pick the smallest precision that preserves acceptable accuracy for your task, then confirm the resulting GB figure leaves headroom for context length and batch size on your target GPU.

Using the parameter count calculator takes a few seconds once you have a model's configuration values, which usually live in a config.json file on Hugging Face or in a published model card.

1. Pick the architecture family. Choose LLaMA/Mistral if the model uses SwiGLU (three FFN matrices) or GPT/Standard if it uses a classic two-matrix MLP. This changes the feed-forward parameter count by 50 percent.
2. Enter the hidden size (d_model). This is the model's main width, listed as hidden_size in most configs.
3. Enter the number of layers (the depth, often num_hidden_layers) and the number of attention heads (used for context, since the QKVO projections are square).
4. Enter the vocabulary size (vocab_size) and the intermediate size (intermediate_size or FFN dimension).
5. Read the results. The headline shows total parameters in billions; the cards show FP16 and INT4 memory; and the breakdown panels split parameters by component and show memory across all four precisions.

If your estimate is slightly higher than the published figure, the difference is almost always tied embeddings: real models often share the input and output embedding matrices, so subtract one copy of V × d to get the tied total.