What is GELU and why does it outperform ReLU in transformer models?

How to think about it

GELU, or Gaussian Error Linear Unit, is the activation function GELU(x) = x · Φ(x), where Φ(x) is the cumulative probability that a standard normal random variable is less than x. It often beats ReLU in Transformer feed-forward layers because it replaces ReLU’s hard cutoff with a smooth, input-dependent gate. That said, “outperforms” is an empirical result, not a mathematical guarantee.

What GELU computes

ReLU is simple:

ReLU(x) = max(0, x)

A negative input becomes exactly zero. A positive input passes unchanged.

GELU instead scales the input by Φ(x):

GELU(x) = x · Φ(x)

For a large positive input, Φ(x) is close to one, so GELU behaves almost like the identity function. For a large negative input, Φ(x) is close to zero, so GELU strongly suppresses the input. Around zero, it makes a gradual decision rather than flipping a switch.

The exact function can also be written using the error function, but implementations commonly use this fast approximation:

GELU(x) ≈ 0.5 · x · (1 + tanh(√(2/π) · (x + 0.044715 · x³)))

The approximation is close enough for normal neural-network use. In PyTorch, torch.nn.functional.gelu(z, approximate="none") requests the exact form, while approximate="tanh" requests the approximation.

Here is the difference at a few inputs:

Input	ReLU	Exact GELU	GELU gradient
-2.0	0.0000	-0.0455	-0.0852
-0.5	0.0000	-0.1543	0.1325
0.0	0.0000	0.0000	0.5000
0.5	0.5000	0.3457	0.8675
2.0	2.0000	1.9545	1.0852

The important row is often the one at -0.5. ReLU outputs zero and sends no gradient backward. GELU outputs a small negative value and still has a useful positive gradient.

For exact GELU, the derivative is:

GELU'(x) = Φ(x) + x · φ(x)

where φ(x) is the standard normal probability density. At zero, the derivative is 0.5. ReLU has a sharp corner at zero: its derivative is zero on the negative side and one on the positive side. At exactly zero, the mathematical derivative is undefined, so libraries choose a convention.

GELU is also non-monotonic. Its output dips slightly below zero, reaching a minimum near input -0.75 and output about -0.17, before rising toward zero as the input becomes more negative. This is why the function is not simply “a smoother ReLU.”

Common misconception: the Gaussian in GELU does not mean the model samples a random gate during every forward pass. Ordinary GELU is deterministic. The stochastic description is an interpretation: if B is a Bernoulli variable with probability Φ(x), then the expected value of x · B is x · Φ(x). The implementation computes that expectation directly; it does not draw B.

Why that can help in a Transformer

The activation usually appears in the feed-forward network, or FFN, inside each Transformer block:

FFN(h) = W₂ · GELU(W₁ · h + b₁) + b₂

Attention mixes information between tokens. The FFN then transforms each token representation independently, feature by feature. In BERT-base, for example, a token vector has 768 hidden values, expands to 3072 intermediate values, passes through GELU, and contracts back to 768 values.

Imagine one of those 3072 intermediate pre-activations is -0.5.

With ReLU:

• the intermediate value becomes 0;
• the gradient through that channel becomes 0;
• that channel receives no learning signal from this example.

With GELU:

• the intermediate value is about -0.154;
• the gradient is about 0.133;
• the channel remains weakly active and can still adjust its weights.

That distinction matters because Transformer representations are continuously changing. A feature that is slightly below zero is not necessarily useless. It may become useful after one update, for another token, or in another layer. ReLU treats the entire negative half of the real line identically. GELU distinguishes -0.1 from -3.0.

The smoothness matters for the same reason. GELU’s gradient changes continuously as the input crosses zero. A small change in a pre-activation produces a small change in the output and gradient. ReLU introduces a hard boundary where the gradient changes abruptly. This does not make the whole neural-network loss surface magically easy, but it removes one avoidable kink from every FFN channel.

GELU’s design also has a distributional motivation. Under common initialization, a linear layer sums many weighted inputs, and those sums can be roughly Gaussian. Transformer blocks also commonly use layer normalization, which keeps hidden representations on a controlled scale. A Gaussian-CDF gate is therefore a reasonable soft threshold for the values an FFN may see.

That is a motivation, not a requirement. GELU does not require Gaussian inputs. After training, the pre-activation distribution can be skewed, heavy-tailed, or very different from a standard normal distribution. The function still works because Φ(x) is simply a fixed smooth gating curve.

There is also a less dramatic but important systems point: Transformer blocks have residual connections. The residual path gives gradients a route around the FFN, so ReLU does not usually kill the entire Transformer when one channel is negative. This is why saying “GELU solves vanishing gradients” is too strong. Its benefit is more local: it keeps more FFN channels responsive and changes their outputs smoothly.

“Outperforms” needs qualification

The original Transformer architecture used ReLU in its feed-forward network. GELU became especially associated with BERT and GPT-style models, and later work found it to be a strong default for large language models. The historical lesson is not that attention requires GELU. It is that, in particular architectures and training regimes, GELU often gives better validation loss or downstream accuracy than ReLU.

There is no universal accuracy premium. A reported improvement depends on:

• model size and FFN width;
• optimizer, learning-rate schedule, and initialization;
• normalization and residual placement;
• pretraining data and training duration;
• whether the comparison uses the same parameter count and throughput.

A claim such as “GELU always improves accuracy by one percent” is not a serious answer. It confuses one experiment with a law of neural networks.

The modern comparison is often not GELU versus ReLU anyway. Many decoder-only language models use gated activations such as SwiGLU. A typical SwiGLU FFN is:

SwiGLU(h) = SiLU(hWg) ⊙ (hWv)

Here SiLU(x) = x · sigmoid(x), Wg creates a learned gate, Wv creates a value branch, and ⊙ means elementwise multiplication. SwiGLU is a related gated design, not merely GELU with a new name. It uses additional projections and usually changes the FFN width, so a fair comparison must account for parameters, memory, and compute.

Trade-offs and production choices

GELU costs more than ReLU. ReLU needs a comparison and a selection. Exact GELU uses an error-function calculation, and the tanh approximation still needs several arithmetic operations. On a large GPU, the matrix multiplications usually dominate runtime and a fused GELU kernel can make the difference modest. On a small CPU model, tiny batch, or strict tail-latency service, the elementwise cost can be visible.

GELU also produces very few exact zeros. ReLU can create activation sparsity, which may help specialized sparse kernels or reduce some downstream work. GELU’s negative outputs are usually small, but they are not zero. If sparsity is a major systems objective, ReLU may be the better engineering choice.

In production, use the activation expected by the architecture and checkpoint. If training from scratch, compare alternatives under the same data, token budget, parameter budget, and wall-clock budget. Measure both model quality and throughput. If the model is already validated with GELU, switching to ReLU solely because it is cheaper is not a harmless refactor.

For a broader comparison with ReLU, SiLU, and other activations, see activation functions.

A failure mode worth recognizing

A common mistake is replacing GELU with ReLU inside a pretrained BERT-style model and reusing the old checkpoint. The first symptom is usually an immediate jump in validation loss or nonsensical outputs, not a slow degradation over six weeks. The weights were learned around GELU’s output scale and gradients; changing the activation changes the function those weights implement.

Another failure appears during training from scratch: the FFN pre-activations become strongly negative in many layers. You may see activation histograms concentrated below -3, tiny FFN gradient norms, and a loss curve that plateaus. GELU does not make badly scaled inputs healthy. Inspect normalization order, biases, initialization, and the statistics of W₁h + b₁ before blaming the activation.

What they’ll ask next

Is GELU always better than ReLU?

No. GELU is often a strong choice for large Transformer FFNs, but ReLU can win on small models, sparse workloads, or latency-constrained systems. The answer depends on both optimization results and serving costs.

Does GELU require the inputs to be Gaussian?

No. The Gaussian CDF is part of GELU’s design, not an assumption the implementation checks. Roughly Gaussian pre-activations helped motivate the function, but GELU accepts any real-valued input.

Is SwiGLU just a better GELU?

Not exactly. SwiGLU adds a learned gate and a second projection, so it changes the architecture, parameter count, and compute pattern. It may improve language-model quality, but it should be compared against GELU with matched resources rather than treated as a drop-in activation swap.

Say this in the interview: “GELU is x · Φ(x), a smooth input-dependent gate that preserves useful gradients around zero; it often beats ReLU in Transformer FFNs, but the gain is empirical and must be weighed against extra compute and the model’s architecture.”