Learning Rate Finder

The learning rate finder is a planning tool that estimates a sensible starting learning rate (LR) for training or fine-tuning a neural network, along with a search range, warmup schedule and weight decay. The learning rate is arguably the single most important hyperparameter in deep learning: it controls how large a step the optimizer takes down the loss landscape on every update. Set it too high and the loss explodes or oscillates and the model diverges; set it too low and training crawls, wastes GPU hours, and may settle in a poor local region. This learning rate calculator turns four practical inputs — task type, model size in billions of parameters, batch size, and optimizer — into a concrete recommendation you can drop straight into your training config.

Rather than relying on a single magic number, the calculator anchors each scenario to a community-validated base learning rate and then scales it. Full fine-tuning of large language models is conservative (around 2e-5), pre-training from scratch is more aggressive (around 1e-4), LoRA / PEFT adapters tolerate higher rates (around 1e-4 because only a small fraction of weights update), and RLHF / DPO alignment is extremely delicate (around 1e-6 to avoid destroying a carefully tuned policy). On top of that base value the tool applies a model-size correction and a batch-size correction so that the recommended optimal learning rate tracks the way real training runs behave.

Because the output is a starting point and a range — not a guarantee — the finder pairs naturally with an empirical LR-range test, where you sweep the learning rate upward over a few hundred steps and watch where the loss curve bottoms out before climbing again. Use this AI training calculator to seed that sweep with a defensible value, saving you from blindly guessing and from the costly divergence that follows an over-large step.

The calculator multiplies a task-specific base learning rate by three adjustment factors and then derives a search range, warmup, and weight decay. Each factor maps directly to a well-known scaling heuristic used by practitioners training transformers and other deep networks.

First, the size multiplier reflects the observation that larger models are more sensitive and need smaller steps. It scales as the square root of the ratio between a 7-billion-parameter reference and your model: sqrt(7e9 / params). A 7B model gets a multiplier of 1.0, a 70B model gets roughly 0.32, and a 1B model gets about 2.65. Second, the batch multiplier follows the linear scaling rule — doubling the effective batch size roughly doubles the gradient signal-to-noise, so you can afford a proportionally larger step. It is simply batch / 32. Third, the optimizer multiplier accounts for the fact that adaptive optimizers like AdamW, Adam, and Adafactor work on a normalized gradient scale (multiplier 1.0), plain SGD with momentum needs about 10x larger rates, and the Lion optimizer prefers roughly 10x smaller rates (multiplier 0.1).

The search range frames the recommendation: the minimum is one tenth of the recommended LR and the maximum is three times it, giving a sweep band for an empirical test. The training schedule is computed from your dataset: steps per epoch equal the dataset size divided by the batch size (rounded up), total steps assume three epochs, and warmup spans the first 6% of total steps. Finally, weight decay defaults to 0.01 when AdamW is selected (decoupled weight decay is its defining feature) and 0 otherwise.

Each input on the learning rate finder shapes the recommendation in a predictable way. Knowing what each does lets you reason about the result instead of treating it as a black box.

The outputs include the recommended learning rate, a min-max range to sweep, warmup steps (about 6% of total steps), weight decay, and a derived training schedule with steps per epoch and total steps across three epochs. Read the recommended LR as a center point: begin there, watch the loss for the first few hundred steps, and nudge toward the maximum if learning is sluggish or toward the minimum if the loss spikes. The range exists precisely because the ideal value is dataset- and architecture-dependent, and a short sweep within these bounds almost always lands near the sweet spot.

A good learning rate is only half the story; how you schedule it over training matters just as much. The learning rate finder recommends a warmup phase covering roughly the first 6% of total steps. During warmup the LR ramps linearly from near zero up to the recommended value, which prevents the large, noisy gradients of early training from destabilizing freshly initialized or freshly unfrozen weights. This is standard practice for transformer training and is especially important for adaptive optimizers whose second-moment estimates are unreliable in the first few hundred updates.

After warmup, most modern runs follow a cosine annealing (cosine decay) schedule that smoothly lowers the LR from its peak down toward a small floor by the end of training. Cosine decay tends to outperform step decay and constant schedules because it spends more time at moderate rates early and gently fine-tunes near the end. The calculator also surfaces weight decay: 0.01 for AdamW, whose decoupled weight decay regularizes weights independently of the gradient update, and 0 for optimizers where coupling weight decay into the gradient would interact badly with adaptive scaling.

Putting it together, a typical recipe from this AI training calculator looks like: linear warmup for ~6% of steps to the recommended LR, cosine decay for the remainder, AdamW with weight decay 0.01, and the min-max range available if you need to retune. If your loss diverges, halve the LR or extend warmup; if it plateaus too early, raise the LR toward the maximum or lengthen the cosine tail. These levers, guided by the finder's numbers, cover the vast majority of practical tuning situations.

The learning rate determines whether an expensive training run succeeds or burns GPU hours on a diverging or stalled model. A single failed multi-GPU fine-tuning run can cost hundreds of dollars in cloud compute, and an LR that is merely suboptimal — not catastrophic — can leave several points of accuracy on the table that no amount of extra data recovers. Using a learning rate calculator to seed a principled starting value and a tight search range is one of the highest-leverage habits in applied deep learning.

Consider the asymmetry: searching for a learning rate is cheap, but a bad choice is expensive. An LR-range test costs a few hundred steps; a divergence discovered after twelve hours of training costs a full run. By anchoring to validated base rates and applying the square-root size rule, the linear batch rule, and optimizer-aware scaling, this finder gives you a number that is rarely more than a factor of two or three away from optimal — close enough that a brief sweep nails it. That is the difference between guessing and engineering.

The tool is equally useful for newcomers and experienced practitioners. Beginners get a safe default that avoids the classic mistake of copying a learning rate from an unrelated tutorial. Experts get a quick sanity check before committing cluster time, and a consistent baseline when comparing optimizers or model sizes. Whether you are doing LoRA fine-tuning on a single GPU, full fine-tuning a 13B model, or aligning a policy with RLHF, starting from a calibrated optimal learning rate keeps your experiments fast, reproducible, and economical.