ReLU Calculator

The ReLU calculator evaluates the Rectified Linear Unit activation function and its most popular variants for any input value x, returning both the output f(x) and its derivative f'(x). ReLU is the most widely used activation function in modern deep learning, and this tool lets you compute it instantly for a single value or for an entire batch of comma-separated inputs at once. Understanding how a neuron transforms its pre-activation into a final output is essential for debugging neural networks, choosing activations, and building intuition about gradient flow.

This ReLU activation calculator supports six variants used across deep learning: standard ReLU, Leaky ReLU, PReLU (parametric ReLU), ELU (Exponential Linear Unit), SELU (Self-Normalizing ELU), and ReLU6. For Leaky ReLU, PReLU, and ELU you also supply an alpha parameter that controls the behavior of negative inputs. The tool reports the activation output, the derivative used during backpropagation, whether the neuron is active or inactive, and the closed-form formula for the variant you selected.

Why does the derivative matter so much? Training a neural network with backpropagation requires the gradient of every activation. During the backward pass, the chain rule multiplies the derivative of each activation by the incoming gradient. ReLU is popular precisely because its derivative is exactly 1 for positive inputs, so gradients pass through the active region without shrinking — sidestepping the vanishing gradient problem that plagued sigmoid and tanh networks. The trade-off is that standard ReLU has a derivative of exactly 0 for negative inputs, which can cause "dying" neurons. The variants in this calculator each offer a different remedy. Use this ReLU calculator to compare them side by side and see exactly how each one treats negative and positive values.

Every variant in this ReLU calculator maps a real-valued input to an output, and each has a corresponding derivative used during training. The table below lists the exact forward formulas the calculator evaluates in JavaScript, where α is the alpha parameter and SELU uses the fixed constants λ = 1.0507 and α_SELU = 1.6733.

Standard ReLU simply clips negative inputs to zero, which is fast and effective but discards all gradient on the negative side. Leaky ReLU and PReLU share the same forward formula — they multiply negative inputs by a small slope α — but PReLU treats α as a learnable parameter optimized during training, whereas Leaky ReLU keeps it fixed (commonly 0.01). ELU replaces the hard negative slope with a smooth exponential curve that saturates toward −α, pushing mean activations closer to zero. SELU scales ELU by the carefully derived constant λ = 1.0507 with α_SELU = 1.6733 so that, under the right initialization, activations self-normalize toward zero mean and unit variance across layers. ReLU6 caps the output at 6, which keeps activations in a fixed low-precision-friendly range that is ideal for mobile and embedded inference.

Picking the right activation among the six options in this ReLU variants calculator depends on your architecture, the depth of your network, and the training stability you observe. There is no universally best choice, but there are clear, well-established guidelines.

Standard ReLU is the default for most convolutional and fully connected hidden layers. It is cheap to compute, plays well with modern initialization, and rarely needs tuning. The main failure mode is the dying ReLU problem: if a neuron's weights push its pre-activation persistently negative, its output and gradient both become zero and it stops learning. Plug a few representative negative values into the calculator and watch the derivative drop to 0 to see this directly.

Leaky ReLU fixes dying neurons by leaking a small gradient through negative inputs. A typical α is 0.01, which keeps a slope of 0.01 on the negative side so the unit can always recover. PReLU takes this one step further by learning α per-channel during training, which can boost accuracy on large image datasets at the cost of a few extra parameters.

ELU swaps the linear negative slope for a smooth exponential that saturates at −α, producing negative outputs that nudge the mean activation toward zero and often speed up convergence. SELU is ELU with fixed scaling constants designed for self-normalizing networks; it works best with the LeCun-normal initialization and alpha-dropout, and can remove the need for batch normalization in deep fully connected stacks. ReLU6 bounds the activation to the interval [0, 6], which is the standard choice in MobileNet and other quantization-friendly architectures because the fixed range maps cleanly onto 8-bit integers. A practical workflow: start with ReLU, switch to Leaky ReLU or ELU if you observe many dead units, and reach for ReLU6 only when you are targeting low-precision mobile inference.

The reason this ReLU calculator reports a derivative alongside every output is that the derivative is what actually drives learning. During backpropagation, the gradient flowing back into a neuron is multiplied by that neuron's activation derivative. For standard ReLU the derivative is a clean step function: exactly 1 wherever the input is positive and exactly 0 wherever it is negative. This binary behavior is both ReLU's greatest strength and its central weakness.

On the positive side, a derivative of 1 means gradients pass through unattenuated. Stack many ReLU layers and the gradient does not shrink the way it would with sigmoid, whose derivative peaks at just 0.25. This is the core reason ReLU enabled the training of very deep networks and largely solved the vanishing gradient problem for feedforward and convolutional models.

On the negative side, a derivative of 0 means no gradient flows at all. If a neuron lands in the negative region for every training example, it receives zero gradient, its weights never update, and it is effectively dead. This is the dying ReLU problem, and it is the motivation for nearly every variant. Leaky ReLU and PReLU keep the negative derivative at α instead of 0; ELU keeps it at αe^x, which is always positive; and SELU keeps it at λα_SELUe^x. ReLU6 reintroduces a second zero-gradient region above x = 6, so its derivative is 1 only on the open interval (0, 6). Use the batch-input field to feed a spread of negative and positive values and confirm which variant preserves gradient signal where your network actually operates — that hands-on comparison is the fastest way to diagnose stalled training.

This ReLU calculator is built for quick experimentation, so you can go from a question to a verified answer in seconds. Start by entering an input value (x) in the first field — this is the pre-activation, the weighted sum your neuron produces before the nonlinearity is applied. The large result card shows the activation output immediately, along with a clear ACTIVE or INACTIVE label based on whether the output is greater than zero.

Next, pick a ReLU variant from the dropdown: Standard ReLU, Leaky ReLU, PReLU, ELU, SELU, or ReLU6. When you select Leaky ReLU, PReLU, or ELU, an alpha field appears so you can tune the negative-region behavior. SELU and ReLU6 use fixed internal constants and therefore do not expose alpha. The formula panel updates live to show the exact closed-form expression for your current selection, and a Derivative card reports the gradient at your chosen input.

To evaluate a whole layer at once, use the batch input field and enter comma-separated numbers such as -2,-1,0,1,2. The calculator parses each value, applies the selected activation, and lists every result with an Active or Off badge so you can scan an entire vector at a glance. This is the fastest way to visualize how an activation reshapes a distribution of pre-activations — for example, watching standard ReLU zero out everything below the threshold while Leaky ReLU preserves a faint negative tail. Combine the single-value view, the derivative card, and the batch list to build a complete picture of how your chosen activation behaves across the input range your model produces.