Transformer Layer Calculator

Calculate transformer architecture parameters and requirements.

Architecture Configuration

Architecture Type

Hidden Size (d_model)

Number of Layers

Number of Attention Heads

Intermediate Size (FFN)

Vocabulary Size

Max Sequence Length

Total Parameters

108.8M

108,805,632 parameters

🎯Head Dimension

⚡GFLOPs/Forward

77.31

Parameters per Layer7.08M

Attention Parameters2.36M

FFN Parameters4.72M

Embedding Parameters23.83M

Activation Memory/Layer1.50 MB

Total Activation Memory18.00 MB