VAEs from scratch

The generative-models overview introduced the VAE as an encoder–decoder trained with a reconstruction loss plus a KL term. That’s the what. This lesson is the why it works: a VAE maximizes the ELBO (a tractable lower bound on the data likelihood), and it’s only trainable at all because of the reparameterization trick.

Why a lower bound

We’d love to maximize log p(x) — make the model assign high probability to real data. But p(x) = ∫ p(x|z) p(z) dz integrates over every possible latent z, which is intractable. The VAE’s move is to introduce an encoder q(z|x) that guesses which z likely produced x, and to maximize a lower bound on log p(x) instead — the Evidence Lower BOund (ELBO):

ELBO = E[ log p(x|z) ]  −  KL( q(z|x) ‖ p(z) )
       \_____________/      \_________________/
        reconstruction          regularizer

Maximizing the ELBO pushes log p(x) up. Its two terms are exactly the VAE loss: a reconstruction term (the decoder should rebuild x from z) and a KL term that keeps the encoder’s latent distribution close to a simple prior p(z) = N(0, 1), so the latent space stays smooth and samplable.

The reparameterization trick

Here’s the snag that nearly sinks the whole approach. Training needs gradients to flow from the loss back through z into the encoder — but z is sampled from q(z|x) = N(μ, σ²), and sampling is not differentiable. You can’t backprop through a random draw.

The trick: move the randomness out of the gradient path. Instead of drawing z directly, draw a fixed noise ε ~ N(0, 1) and compute z = μ + σ · ε. Now z is a deterministic, differentiable function of μ and σ (the encoder’s outputs), with ε as an external constant — so gradients flow straight through to the encoder:

import numpy as np
rng = np.random.default_rng(0)

mu, sigma = 2.0, 0.5
eps = rng.standard_normal(5)              # external randomness, no gradient needed
z = mu + sigma * eps                      # differentiable w.r.t. mu and sigma

print("eps   :", eps.round(2))
print("z     :", z.round(2), " (= mu + sigma*eps)")
print("dz/dmu = 1 ;  dz/dsigma = eps =", eps.round(2))

kl = 0.5 * (mu**2 + sigma**2 - np.log(sigma**2) - 1)   # closed-form KL(N(mu,sigma^2) || N(0,1))
print(f"KL(N({mu},{sigma}^2) || N(0,1)) = {kl:.3f}")

eps   : [ 0.13 -0.13  0.64  0.1  -0.54]
z     : [2.06 1.93 2.32 2.05 1.73]  (= mu + sigma*eps)
dz/dmu = 1 ;  dz/dsigma = eps = [ 0.13 -0.13  0.64  0.1  -0.54]
KL(N(2.0,0.5^2) || N(0,1)) = 2.318

Because z = μ + σ·ε, the derivatives are trivial — ∂z/∂μ = 1 and ∂z/∂σ = ε — so the gradient reaches μ and σ. And the KL term has a closed form for two Gaussians (KL = ½ Σ(μ² + σ² − log σ² − 1)), so no sampling is needed for the regularizer at all. Those two facts are what make the ELBO differentiable end-to-end.

Why VAE samples are blurry

With the ELBO trainable, generation is easy: sample z ~ N(0, 1), decode. But the reconstruction term averages over many images consistent with a latent point — pixel-wise MSE rewards the mean of plausible outputs, and the mean of several sharp images is a soft, blurry one. That’s the structural reason VAEs trade sharpness for their smooth, well-behaved latent space (the opposite trade from a GAN).

In one breath

• A VAE can’t maximize the intractable log p(x) = ∫ p(x|z)p(z)dz, so it maximizes the ELBO, a tractable lower bound, using an encoder q(z|x).
• The ELBO = reconstruction − KL: rebuild x from z, while keeping the latent close to a simple prior N(0,1) so the space stays smooth and samplable.
• Training needs gradients through z, but sampling isn’t differentiable — the reparameterization trick writes z = μ + σ·ε (ε ~ N(0,1)), making z differentiable in μ, σ (∂z/∂μ=1, ∂z/∂σ=ε).
• The Gaussian KL is closed-form (½Σ(μ²+σ²−log σ²−1)), so the regularizer needs no sampling — together these make the ELBO differentiable end-to-end.
• The reconstruction loss averages, so VAE samples are blurry — the trade for a smooth latent; the ELBO + reparam template also powers latent diffusion, β-VAE, and molecular design.

Quick check

Next

The VAE’s adversarial counterpart — sharp but unstable — is GANs from scratch; both are framed in the generative-models overview. The Gaussian/KL machinery comes from distributions.