datarekha
Deep Learning Medium Asked at GoogleAsked at MetaAsked at Mistral

What is SwiGLU and why did modern LLMs replace the ReLU/GELU MLP with it?

For ML EngineerAI / LLM Engineer
The short answer

SwiGLU is a gated feed-forward layer: it projects the input into two paths, passes one (the gate) through SiLU, multiplies the two elementwise, then projects down — SiLU(x·W_gate) ⊙ (x·W_up) · W_down. The elementwise gate is a smooth, learned, per-feature volume control, so the network can decide how much of each feature to pass rather than a hard ReLU on/off. It gives better quality per parameter, which is why LLaMA, Mistral, and Qwen use it; to keep the parameter budget equal it uses a smaller (~2/3) hidden width since it has three weight matrices instead of two.

How to think about it

The original transformer FFN was Linear → ReLU → Linear. SwiGLU adds a gating path:

SwiGLU(x) = ( SiLU(x · W_gate) ⊙ (x · W_up) ) · W_down
  • W_up is the value path; W_gate is the gate path; SiLU is x·σ(x).
  • The elementwise product ⊙ is the key: the gate scales each feature of the value path continuously.

Why it helps: a ReLU can only pass a feature or kill it. The gate is a smooth, learned, per-feature multiplier — where SiLU(gate) ≈ 0 the feature is suppressed regardless of its value, where it opens the feature passes and can be amplified. Empirically this improves quality per parameter on language modeling.

Learn it properly Anatomy of a modern LLM

Keep practising

What is GELU and why does it outperform ReLU in transformer models? Compare sigmoid, tanh, ReLU, leaky ReLU, and GELU — when would you pick each? How do LSTM gates solve the vanishing gradient problem? What is RMSNorm and why do modern LLMs like Llama use it instead of LayerNorm? What is GGUF, and what does a quantization tier like Q4_K_M mean?
All Deep Learning questions

Explore further

Activation functions What an LLM is Vision-language model architectures
ReLU Sigmoid Vanishing Gradient LSTM
Skip to content