datarekha
Section 4 chapters · 28 of 28 lessons

Deep Learning

Neural networks the way you'll build them today — PyTorch, autograd and backprop from scratch, the training stack that actually matters, CNNs and RNNs, and deep generative models (VAEs, GANs, diffusion). Language models and transformers live in their own NLP & Transformers section.

The Deep Learning journey 0 / 28 completed
  1. Chapter 01

    Foundations

    7 lessons
  2. 01 PyTorch quickstart Tensors, autograd, and the canonical training loop — the five steps of PyTorch you'll write a thousand times. Beginner8 min
  3. 02 Autograd The computation graph, requires_grad, and the magic behind loss.backward() — demystified by building a tiny autograd from scratch. Intermediate7 min
  4. 03 Backprop by hand backward() is not magic — it's the chain rule applied mechanically over a graph. Build a 30-line autograd engine from scratch and watch gradients f… Intermediate9 min
  5. 04 The training loop Forward, loss, backward, step, zero-grad — the five-line ritual that turns a pile of layers into a model that learns. Built and run from scratch. Beginner8 min
  6. 05 Activation functions ReLU, GELU, SiLU, sigmoid, softmax — what each one does, what it's for, and why a network without them is just a linear regression in disguise. Beginner5 min
  7. 06 Softmax How a network turns raw scores into a probability distribution: the exp-over-sum formula, why exp, the subtract-the-max stability trick, and how te… Intermediate7 min
  8. 07 Loss Functions The loss is the only thing your network actually optimizes. Pick the wrong one and training chases the wrong goal — here is how to choose. Intermediate8 min
  9. Chapter 02

    Training

    11 lessons
  10. 08 Weight initialization Set the starting weights too big and the signal explodes; too small and it vanishes. Why Xavier and He initialization are the scales that let deep… Intermediate7 min
  11. 09 Vanishing & exploding gradients Backprop multiplies a long chain of numbers. Get them slightly below 1 and gradients vanish; slightly above and they explode. How to diagnose it wi… Intermediate7 min
  12. 10 SGD → Adam → AdamW How the update rule turns gradients into weight changes — and why AdamW is the default you should reach for today. Intermediate7 min
  13. 11 Learning-rate schedules Cosine decay, warmup, step decay, ReduceLROnPlateau — when to decay your learning rate and why warmup is mandatory for transformers. Intermediate6 min
  14. 12 Batch size ↔ learning rate Batch size and learning rate are a coupled pair, not two independent knobs. The linear scaling rule, warmup, and gradient accumulation — how to tra… Intermediate6 min
  15. 13 Overfitting & bias–variance A model that nails the training data but fails on new data has memorized noise. The bias–variance tradeoff, the train/validation curve, and early s… Beginner7 min
  16. 14 Dropout, BN, LN The three regularization and stabilization tricks every modern network uses — what each does, why it works, and when to reach for which. Intermediate7 min
  17. 15 Mixed precision Train in fp16 or bf16 instead of fp32 — half the memory, often 2x faster, no accuracy loss. The autocast + GradScaler pattern. Advanced5 min
  18. 16 Activation checkpointing Understand what training VRAM contains, why activations grow with batch and sequence length, and how PyTorch and Transformers recompute selected fo… Advanced11 min
  19. 17 Multi-GPU: DDP & FSDP When the model fits on one GPU, replicate it (DDP). When it doesn't, shard it (FSDP). The memory math behind the decision, and how data parallelism… Advanced8 min
  20. 18 Pipeline parallelism When a model is too big to fit on one GPU, you split it across GPUs by layer — an assembly line of stages. The catch is the 'pipeline bubble' of id… Advanced9 min
  21. Chapter 03

    Architectures

    2 lessons
  22. 19 Convolutional neural networks How CNNs see — convolution, kernels, feature maps, stride, padding, and pooling — built from scratch in NumPy, with an interactive filter you scan… Intermediate10 min
  23. 20 RNNs & LSTMs Before transformers, networks read text one word at a time, carrying a memory. How recurrent networks work, why their gradients vanish over long se… Intermediate8 min
  24. Chapter 04

    Generative models

    8 lessons
  25. 21 GANs, VAEs & Diffusion Discriminative models tell cats from dogs; generative models can draw a new cat. Three families do it three different ways — and one of them now po… Advanced9 min
  26. 22 VAEs from scratch The generative-models overview said a VAE encodes to a latent and decodes, trained with reconstruction plus KL. This is the math underneath: maximi… Advanced8 min
  27. 23 GANs from scratch The overview said a GAN pits a generator against a discriminator. This is the math: the minimax objective, the alternating training loop, the non-s… Advanced8 min
  28. 24 Diffusion models (DDPM) The generative-models overview said diffusion adds noise forward and learns to reverse it. This is the math: the forward process has a closed form… Advanced8 min
  29. 25 Latent diffusion & Stable Diffusion DDPM denoises in pixel space, which is expensive. Latent diffusion — the architecture behind Stable Diffusion — runs the whole process inside a VAE… Advanced8 min
  30. 26 Flow matching & rectified flow Diffusion's reverse path is a curved, stochastic, thousand-step trajectory. Flow matching asks a simpler question: why not learn a straight path fr… Advanced7 min
  31. 27 Evaluating generative models How do you score a generator when there's no right answer? Accuracy is meaningless for image generation. You need metrics for quality (do samples l… Intermediate7 min
  32. 28 Video, 3D & audio generation Image generation is largely solved; the frontier extends the same diffusion and flow machinery along three new axes. Video adds time and needs temp… Advanced7 min
  33. End of section 0 / 28 complete

    Make it stick — pass every quiz.

    Each lesson has a short quiz at the bottom. Passing the quiz is what marks the lesson complete and counts toward your certificate.

    Section complete 28 / 28 lessons

    Nice work — you finished Deep Learning.

    Certificates are earned per learning path, not per section. Here's where this section takes you:

FAQCommon questions

Deep Learning — frequently asked questions

Straight answers to the questions people ask most about deep learning.

What does PyTorch's autograd actually do?

Autograd records the operations in your forward pass as a graph, then automatically computes gradients of the loss with respect to every parameter by applying the chain rule backward. That's what lets you train with `loss.backward()` without deriving gradients by hand.

Why do neural networks need activation functions?

Without a non-linear activation, stacking layers just composes linear functions, which collapses to a single linear model no matter how deep. Activations like ReLU add non-linearity, letting the network approximate complex, curved decision boundaries.

What's the difference between SGD, Adam, and AdamW?

SGD updates weights using the raw gradient (optionally with momentum). Adam adapts the step size per parameter using running estimates of the gradient and its variance, often training faster. AdamW fixes how Adam handles weight decay for better regularisation, and is the common default for transformers.

What loss function should I use?

Use cross-entropy for classification — it heavily penalises confident wrong predictions — and mean squared error (or a robust variant like Huber) for regression. The loss must match the output layer, for example softmax outputs paired with cross-entropy.

What is a transformer and why did it change everything?

A transformer is an architecture built on self-attention, which lets every token directly weigh every other token regardless of distance, and processes a whole sequence in parallel rather than step by step like an RNN. That parallelism and long-range modeling are what made modern large language models possible.

Read the lesson
Skip to content