What is SwiGLU and why did modern LLMs replace the ReLU/GELU MLP with it?

How to think about it

Short answer

SwiGLU is a gated feed-forward network, or FFN, inside a transformer block. It sends the token representation through two learned projections, applies SiLU to one path, multiplies the paths element by element, and projects the result back down.

Modern LLMs did not abandon ReLU or GELU because they stopped working. Many adopted SwiGLU because experiments repeatedly found better language-model quality per parameter when the extra projection was paid for by using a narrower hidden layer.

What the layer computes

A standard transformer FFN is a small multilayer perceptron applied independently to each token:

FFN(x) = activation(x · W_up) · W_down

Here, x is one token’s representation, and W_up expands it from the model width to a wider intermediate width. The activation adds nonlinearity. W_down projects the result back to the model width.

SwiGLU adds a second expansion path:

g = x · W_gate
v = x · W_up

SwiGLU(x) = (SiLU(g) ⊙ v) · W_down

W_gate and W_up are separate learned matrices with the same shape. v is the value path: it proposes information to send onward. g is the gate path: it determines how strongly each intermediate channel should participate. The symbol ⊙ means elementwise multiplication, so channel 17 in the gate multiplies channel 17 in the value path.

SiLU is the smooth activation

SiLU(z) = z · sigmoid(z)

The sigmoid function maps a number to a value between zero and one. SiLU is sometimes called Swish.

A useful PyTorch-shaped version, with biases omitted, is:

import torch.nn.functional as F

def swiglu(x, w_gate, w_up, w_down):
    gate = F.silu(x @ w_gate)
    value = x @ w_up
    return (gate * value) @ w_down

The order matters. This is not merely “apply SiLU instead of GELU.” There are two independent projections and a multiplication between them.

A numerical example

Look at one intermediate channel rather than a whole matrix.

Suppose the value path produces v = 3. The gate path produces different values in different contexts:

Gate preactivation g	SiLU(g)	Output SiLU(g) · v
-6	approximately -0.015	approximately -0.045
0	0	0
2	approximately 1.762	approximately 5.285

When the gate is strongly negative, the value is almost suppressed. When the gate is near zero, it is shut off. When the gate is positive, it can pass and amplify the value.

That last detail matters: the gate is not a probability and is not restricted to the interval from zero to one. SiLU can produce a negative coefficient or a coefficient greater than one. “Gate” is the right intuition, but “on or off switch” is too crude.

Compare this with ReLU. ReLU computes max(0, z): a negative hidden value becomes exactly zero, while a positive value passes unchanged. GELU softens that decision, but it still has one projected hidden vector. SwiGLU has a separate learned control signal for every intermediate channel.

The two paths also depend on the same input. That lets the layer behave conditionally. One input pattern can make a channel’s value large but its gate nearly zero. Another pattern can open the gate and pass that value. In effect, the layer can learn interactions of the form “use this transformation when these input conditions are present.”

Why this helps language models

The FFN is not decorative filler between attention layers. Attention mixes information across token positions; the FFN transforms each token position’s resulting representation. It is a large store of learned nonlinear transformations.

A ReLU or GELU FFN says, roughly, “project the token into hidden features, transform every feature according to one activation, then combine them.” SwiGLU says, “produce candidate features and separately decide how much of each candidate is useful for this token.”

That multiplicative interaction gives the network more expressive behaviour at the same broad parameter budget. A hidden channel can respond to a combination of signals rather than only to the magnitude of one projection. For language, that is useful because the same word or phrase can require different transformations depending on syntax, topic, surrounding facts, and position in a sequence.

SiLU also avoids ReLU’s hard zero boundary and is smooth everywhere. Smoothness can make optimization friendlier, but it is not the whole explanation: GELU is smooth too. The important architectural change is the second projection and the learned multiplication. The advantage is empirical rather than a theorem saying SwiGLU must win on every task.

This is why gated variants became common in systems such as PaLM, LLaMA-family models, Mistral, and Qwen. They tended to produce a lower training loss or better benchmark quality under comparable model-size constraints. The exact gain depends on the dataset, model scale, tokenizer, optimizer, training duration, and what “same budget” means.

The parameter-budget catch

A vanilla FFN has two large matrices. If the model width is d and its intermediate width is m, the weight count is approximately:

2 · d · m

SwiGLU has three:

3 · d · m

If both versions use the same intermediate width, SwiGLU has about 50 percent more FFN weights. That would be an unfair comparison in many scaling experiments.

So implementations usually make the SwiGLU intermediate layer about two-thirds as wide as the vanilla FFN’s layer. Solving

3 · d · m_swiglu ≈ 2 · d · m_vanilla

gives

m_swiglu ≈ 2/3 · m_vanilla

For a concrete example, suppose d = 4096 and a vanilla FFN uses m = 16384.

The two vanilla matrices contain:

2 · 4096 · 16384 = 134,217,728

weights, ignoring biases.

A parameter-matched SwiGLU uses an intermediate width of about 10923, before rounding to a hardware-friendly size:

3 · 4096 · 10923 ≈ 134 million

So the comparison is not “three full-size projections beat two full-size projections.” It is closer to “two narrower input paths plus one output path often beat a conventional wider FFN at roughly the same weight budget.”

The arithmetic also explains why SwiGLU does not automatically make inference 50 percent slower. With a matched width, total matrix-multiplication work can be similar. Implementations can often combine the gate and value projections into one larger input projection and split the result. Actual latency still depends on kernel fusion, memory traffic, batch size, quantization, and hardware. Parameter arithmetic is not a latency benchmark wearing a fake moustache.

The senior-level nuance

“Modern LLMs replaced GELU with SwiGLU” is too absolute. Many large language models use SwiGLU or another gated activation, but some use GELU, ReLU variants, gated linear units, mixture-of-experts layers, or architectures that do not have a conventional FFN at all.

SwiGLU is also dense. Every token normally computes both projections, even when the gate later makes a channel’s output tiny. The gate controls values; it does not automatically skip the computation. If the goal is actual conditional computation, a mixture-of-experts layer is a different mechanism.

There is a real systems trade-off. SwiGLU needs another weight matrix and two intermediate paths, which can increase memory traffic and activation storage. A small model running on a latency-sensitive device may benefit more from a simpler, highly optimized GELU kernel than from a modest quality improvement. The right comparison must hold the parameter budget, training recipe, and serving constraints steady.

Common trap: SiLU(x · W) ⊙ (x · W) is not the usual SwiGLU layer. The gate and value paths need separate learned projections, W_gate and W_up. Accidentally reusing one projection reduces the model’s capacity and changes the architecture. The first symptom may be a training loss that decreases but remains noticeably worse than a reference run.

What they’ll ask next

“Is SwiGLU just SiLU replacing GELU?”

No. Replacing GELU with SiLU in a two-matrix FFN is a different experiment. SwiGLU adds a second projection and multiplies the SiLU-transformed gate path by the value path.

“What are GEGLU and ReGLU?”

They use the same gated pattern but change the gate activation. GEGLU uses GELU, ReGLU uses ReLU, and SwiGLU uses SiLU. The family is best remembered as:

activation(x · W_gate) ⊙ (x · W_up)

“Does the gate make the model sparse?”

Usually no. It provides dense, continuous per-channel modulation. A gate close to zero reduces a channel’s contribution, but the matrix multiplications still happen unless the implementation adds a separate sparsity mechanism.

Say this in the interview

“SwiGLU is a gated FFN: SiLU transforms one projection, that result gates a second projection elementwise, and a third matrix projects it down; its advantage is usually better quality per parameter, so the intermediate width is reduced to about two-thirds to offset the extra projection.”