GANs from scratch
The overview said a GAN pits a generator against a discriminator. This is the math: the minimax objective, the alternating training loop, the non-saturating loss that fixes vanishing gradients, and the equilibrium where the discriminator can do no better than a coin flip — plus why that game is so hard to balance.
Written and reviewed by Shreyash Prashu Last reviewed
What you'll learn
- The minimax objective and what each network optimizes
- The alternating training loop and the non-saturating generator loss
- The equilibrium — generator matches the data, discriminator hits 0.5
- Why GANs are unstable, and the variant family (DCGAN to StyleGAN)
Before you start
The generative-models overview framed a GAN as a generator versus a discriminator. This lesson writes down the game they’re actually playing — the minimax objective, how you train two competing networks without one crushing the other, and why the whole thing is famously delicate.
The minimax objective
A GAN has two networks with opposite goals over one objective:
min_G max_D E_x[ log D(x) ] + E_z[ log(1 − D(G(z))) ]
\____________/ \___________________/
D: score real high D: score fakes lowThe discriminator D maximizes this — push D(x) toward 1 on real data and D(G(z)) toward 0
on fakes. The generator G minimizes it — make D(G(z)) large, i.e. fool D into scoring its
fakes as real. It’s a zero-sum game: G’s gain is D’s loss. There’s no fixed “looks realistic”
loss written by hand — D is the loss, learned from data and improving as G improves.
Training: alternate, and don’t let G starve
You can’t minimize and maximize at once, so you alternate: take a gradient step to improve D,
then a step to improve G, repeat. One subtlety matters in practice. Early on, G is terrible, D
rejects its fakes confidently (D(G(z)) ≈ 0), and the log(1 − D(G(z))) term saturates — its
gradient vanishes, so G can’t learn. The fix is the non-saturating loss: instead of minimizing
log(1 − D(G(z))), G maximizes log D(G(z)). Same goal (fool D), but a strong gradient exactly
when G is losing.
The equilibrium
At the game’s optimum, G’s distribution equals the real data distribution, and then D is helpless — every input, real or fake, is equally likely, so the best it can do is output 0.5. As G improves, D’s accuracy decays toward that coin flip:
real_mean, gen_mean = 0.0, 3.0
for step in range(1, 6):
gen_mean += 0.5 * (real_mean - gen_mean) # G moves its distribution toward real
gap = abs(gen_mean - real_mean)
d_acc = 0.5 + 0.5 * min(1.0, gap / 3.0) # D: ~1.0 when far apart, 0.5 when matched
print(f"step {step}: gen_mean={gen_mean:+.2f} D accuracy={d_acc:.2f}")step 1: gen_mean=+1.50 D accuracy=0.75
step 2: gen_mean=+0.75 D accuracy=0.62
step 3: gen_mean=+0.38 D accuracy=0.56
step 4: gen_mean=+0.19 D accuracy=0.53
step 5: gen_mean=+0.09 D accuracy=0.52As the generator’s distribution closes on the real one, the discriminator’s accuracy slides from 0.75 toward 0.5 — the signature that the GAN has converged: D can no longer tell real from fake.
Why GANs are so hard — and the variant family
That clean equilibrium is rarely reached smoothly. The minimax is a saddle point, not a minimum, so training can oscillate instead of converging. If D gets too strong too fast, G’s gradient vanishes (the saturation problem); if G finds one output that reliably fools D, it stops exploring — mode collapse. Stabilizing GANs is a whole literature: WGAN (a smoother Wasserstein loss), spectral normalization, and careful balancing of the two networks.
In one breath
- A GAN plays a minimax game:
min_G max_D E[log D(x)] + E[log(1−D(G(z)))]— D scores real high and fakes low; G fools D. D is the learned loss. - You alternate D and G gradient steps; early on G’s loss saturates, so use the
non-saturating form (G maximizes
log D(G(z))) for a strong gradient when G is losing. - At equilibrium, G’s distribution equals the data’s and D outputs 0.5 everywhere — the demo’s D accuracy slides 0.75 → ~0.5 as G matches real.
- GANs are unstable: a saddle point (oscillation), vanishing gradient if D dominates, and mode collapse; fixes include WGAN and spectral normalization.
- The variant family — DCGAN, conditional GAN/CTGAN, pix2pix/CycleGAN, StyleGAN — and the lasting idea that a discriminator is a learned loss reusable beyond images.
Quick check
Quick check
Next
The GAN’s stable-but-blurry counterpart is VAEs from scratch; both sit in the generative-models overview. Conditional GANs for rare-class oversampling connect to class imbalance.
Practice this in an interview
All questionsGANs train a generator and discriminator adversarially to produce sharp samples but suffer from unstable training and mode collapse. VAEs optimise a tractable evidence lower bound for principled probability modelling but generate blurry samples. Diffusion models iteratively denoise from Gaussian noise, achieving state-of-the-art sample quality and diversity at the cost of slow sampling.
Discriminative models learn the conditional distribution P(y|x) directly and focus entirely on the decision boundary; generative models learn the joint distribution P(x,y) and can generate new samples. Discriminative models typically achieve higher classification accuracy with sufficient labeled data; generative models excel when data is scarce, you need to synthesize data, or the problem requires modeling the input distribution.
The vanishing gradient problem occurs when repeated Jacobian and derivative products make gradients shrink toward zero as they travel backward, so early layers barely learn. Common remedies are non-saturating hidden activations, suitable initialization, normalization, residual connections, and architectures with shorter or additive gradient paths.
Adam combines momentum-like gradient averaging with per-parameter scaling based on recent squared gradients. It usually handles noisy, differently scaled gradients and reaches a useful solution faster than vanilla SGD, but it still needs learning-rate and regularization tuning and can generalize worse than well-tuned momentum SGD.