Transformer Layer Calculator

The transformer layer calculator turns an architecture specification into a precise budget of parameters, compute, and memory. Every modern large language model — BERT, GPT, T5, Llama, and their descendants — is built by stacking identical transformer layers, each containing a multi-head self-attention block, a feed-forward network (FFN), and two layer-normalization steps. This calculator takes the dimensions that define those layers (hidden size, number of layers, attention heads, FFN intermediate size, vocabulary size, and maximum sequence length) and returns the parameter count per layer, the total model size, the per-head attention dimension, the floating-point operations (FLOPs) of a forward pass, and the activation memory you will need at run time.

Knowing these figures up front is what separates a training run that fits on your GPUs from one that crashes with an out-of-memory error halfway through the first epoch. When you read that a model has "110 million parameters" or "1.5 billion parameters," that number comes directly from the same arithmetic this tool runs: embedding weights plus the per-layer weights multiplied by the depth of the stack. By exposing the breakdown — attention parameters, FFN parameters, and embedding parameters separately — the transformer layer calculator shows you exactly where the weight budget is being spent and which knob to turn when you want a bigger or smaller model.

The calculator supports the three dominant transformer families through its architecture selector. Encoder-only designs such as BERT process the whole sequence at once and excel at understanding tasks like classification and retrieval. Decoder-only designs such as GPT generate text autoregressively and power most chat and completion models. Encoder-decoder designs such as T5 pair a reader with a writer for translation and summarization. While the core parameter math is shared across all three, the selector helps you frame the configuration you are sizing and compare it against the canonical reference models for each family.

Everything below explains the exact equations the tool evaluates, with worked examples that reproduce the on-screen numbers for BERT-base and GPT-style configurations. Whether you are planning a from-scratch pretraining job, fine-tuning an existing checkpoint, or simply trying to understand how transformer architecture scales, the transformer layer calculator gives you the concrete parameter, FLOP, and memory figures you need to make decisions with confidence.

The transformer layer calculator builds the total parameter count from two pieces: the embedding tables, which exist once, and the per-layer weights, which repeat for every layer in the stack. Start with a single transformer layer. The self-attention block holds four learned projection matrices — query, key, value, and output. In the standard design each is a square matrix of size hidden × hidden, so the query, key, and value projections together contribute 3d² weights and the output projection adds another d², giving 4d² attention parameters per layer.

The feed-forward network is the second large block. It expands the hidden representation up to the intermediate size and then projects it back down, so it uses two matrices of size hidden × intermediate. That contributes 2 × d × ff parameters, where ff is the FFN intermediate width (typically four times the hidden size). Each layer also carries two layer-normalization operations, and each layer norm has a scale and a bias vector of length hidden, contributing 4d small parameters per layer. Summing these gives the parameters per transformer layer reported on screen.

The embedding side is computed once and added on top. The token embedding table maps every vocabulary entry to a hidden vector, so it holds V × d weights, and the position embedding table adds seqLen × d for the maximum sequence length. The grand total is the embedding parameters plus the number of layers times the per-layer parameters. The calculator also derives the head dimension as hidden size divided by the head count, which tells you how wide each individual attention head is — almost always 64 or 128 in well-designed models.

Inputs and their roles

• Hidden Size (d_model): the embedding width that drives nearly every weight count quadratically.
• Number of Layers: the depth of the stack; total layer parameters scale linearly with it.
• Number of Attention Heads: splits the hidden size into parallel heads and sets the head dimension.
• Intermediate Size (FFN): the feed-forward expansion width, usually four times the hidden size.
• Vocabulary Size: the number of tokens, which sizes the token embedding table.
• Max Sequence Length: the longest context, which sizes position embeddings and drives quadratic FLOPs.

Beyond parameters, the transformer layer calculator estimates two run-time costs: the floating-point operations of a forward pass and the activation memory that pass allocates. These are what determine training throughput and whether a configuration fits on your accelerator.

The FLOPs figure adds the cost of each layer's two main blocks. Inside attention, the query, key, value, and output projections cost about 4 × seqLen × d² operations, while the attention score computation over the sequence costs 2 × seqLen² × d. That second term scales with the square of the sequence length, which is the famous quadratic bottleneck of self-attention: doubling the context roughly quadruples this part of the work. The feed-forward block adds 4 × seqLen × d × ff operations. The calculator multiplies the sum of these per-layer FLOPs by the number of layers and divides by one billion to report GFLOPs per forward pass.

The activation memory estimate captures the temporary tensors a forward pass keeps in memory. The calculator models the per-layer activation footprint as seqLen × d × 4 bytes, assuming fp32 (four bytes per value), and multiplies by the layer count for the total. This is a deliberately compact estimate of the dominant hidden-state activations rather than every intermediate buffer, so treat it as a planning baseline. Real training adds attention score matrices, optimizer states, and gradients on top, while switching to bf16 or fp16 halves the per-value cost. The table below summarizes the quantities the calculator computes and how each one scales.

The architecture selector frames which transformer family you are sizing, and although the underlying parameter math is the same, the design choices behind each family shape how you read the results. The three options correspond to the most influential model lineages in natural language processing.

Encoder-only models (BERT-style) read an entire input sequence with bidirectional attention, meaning every token can attend to every other token. This makes them strong at understanding tasks — sentence classification, named-entity recognition, semantic search, and embedding generation. BERT-base, the canonical example, uses a hidden size of 768, twelve layers, twelve heads, and an FFN intermediate size of 3072, landing near 110 million parameters, a figure this calculator reproduces from its default inputs. Decoder-only models (GPT-style) generate text left to right with causal masking, so each token attends only to earlier positions. This autoregressive design powers chat assistants and code generators; scaling the same per-layer formulas to wider hidden sizes and deeper stacks is exactly how GPT-2 grew from 124 million to 1.5 billion parameters.

Encoder-decoder models (T5-style) combine a bidirectional encoder that reads the input with an autoregressive decoder that writes the output, connected by cross-attention. They shine at sequence-to-sequence tasks such as translation and summarization. Because a full encoder-decoder stack carries both towers plus cross-attention, its real parameter count is larger than a single-tower estimate; treat the calculator's per-layer figures as the building block you multiply across both towers. Across all three families the headline lesson is the same: hidden size dominates because attention parameters grow with its square, while depth and FFN width add linearly. Picking the right balance of width, depth, and vocabulary is the core craft of transformer architecture design, and the calculator lets you explore those trade-offs instantly.

Each panel in the transformer layer calculator maps to a concrete engineering decision. The headline total parameters figure, shown in millions or billions, is the number you quote when describing model size and the number that determines how much GPU memory the weights alone consume — roughly two bytes per parameter in bf16, so a 110-million-parameter model needs about 220 MB just for weights, before optimizer states and activations.

The parameters-per-layer breakdown is where you diagnose the shape of your design. If attention parameters dominate, your hidden size is large relative to the FFN; if FFN parameters dominate, your intermediate size is wide. The embedding parameters line is easy to overlook but can be surprisingly large for models with big vocabularies and modest depth — in BERT-base the roughly 24 million embedding parameters are a meaningful slice of the 110 million total. Watch this number when you change vocabulary size, because token embeddings scale linearly with it.

The head dimension output should land on 64 or 128; if it does not, adjust the head count so the hidden size divides cleanly, since odd head dimensions waste GPU tensor-core throughput. The GFLOPs per forward pass value is your compute budget for a single sequence, and if it jumps sharply between two configurations the cause is almost always a longer sequence length feeding the quadratic attention term. Finally, the activation memory estimates tell you the baseline working set per layer and for the whole stack; when that number climbs uncomfortably high, the levers are a shorter context, a smaller batch, gradient checkpointing, or a lower-precision format. Treat every figure here as a well-grounded planning estimate rather than an exact production measurement, because real frameworks add overhead the simplified formulas intentionally leave out.