Skip to content
datarekha

Course · 4 chapters · 39 lessons

Deep Learning

Neural networks the way you'll build them today — PyTorch, autograd and backprop from scratch, the training stack that actually matters, CNNs and RNNs, and deep generative models (VAEs, GANs, diffusion). Language models and transformers live in their own NLP & Transformers section.

The Deep Learning journey 0 / 39 completed
  1. Chapter 01

    Foundations

    8 lessons
  2. 01 PyTorch quickstart A practical PyTorch quickstart: tensors, autograd, devices, dtypes, and the training loop behind every neural network. Beginner8 min
  3. 02 Autograd The computation graph, requires_grad, and the mechanics behind loss.backward() — demystified by building a tiny autograd from scratch. Intermediate7 min
  4. 03 The training loop Forward, loss, backward, step, zero-grad — the five-line ritual that turns a pile of layers into a model that learns. Built and run from scratch. Beginner8 min
  5. 04 Backprop by hand backward() is not magic — it's the chain rule applied mechanically over a graph. Build a 30-line autograd engine from scratch and watch gradients f… Intermediate9 min
  6. 05 Activation functions ReLU, GELU, SiLU, sigmoid, and softmax: what each computes, why nonlinearities make depth useful, and how to match an activation to the task and loss. Beginner5 min
  7. 06 Softmax How a network turns raw scores into a probability distribution: the exp-over-sum formula, why exp, the subtract-the-max stability trick, and how te… Intermediate7 min
  8. 07 Loss Functions The loss is the only thing your network actually optimizes. Pick the wrong one and training chases the wrong goal — here is how to choose. Intermediate8 min
  9. 08 Debugging a training run A symptom-first playbook for finding why a deep-learning run is stuck, unstable, slow, or impossible to reproduce. Intermediate12 min
  10. Chapter 02

    Training

    17 lessons
  11. 09 Weight initialization A deep network can fail before its first optimizer step if its weights start at the wrong scale. Learn why variance and gradients compound, how Xav… Intermediate7 min
  12. 10 Vanishing & exploding gradients A gradient is a long product of local derivatives. See why that product vanishes or explodes, how residuals and initialization change the mechanism… Intermediate7 min
  13. 11 SGD → Adam → AdamW How the update rule turns gradients into weight changes — and why AdamW is the default you should reach for today. Intermediate7 min
  14. 12 Learning-rate schedules How warmup, cosine decay, step decay, and ReduceLROnPlateau change optimization, with the production mistakes that quietly ruin a training run. Intermediate6 min
  15. 13 Batch size ↔ learning rate Batch size and learning rate are coupled because one controls gradient noise and the other controls step size. Learn linear scaling, warmup, gradie… Intermediate6 min
  16. 14 Normalization layers How BatchNorm, LayerNorm, RMSNorm, and GroupNorm reshape activations so deep networks optimize more reliably. Intermediate12 min
  17. 15 Overfitting & bias–variance A model that looks brilliant on its training data can still fail in production. Learn to read the train-validation curve, separate bias from varian… Beginner7 min
  18. 16 Dropout, BN, LN The three regularization and stabilization tricks every modern network uses — what each does, why it works, and when to reach for which. Intermediate7 min
  19. 17 Data augmentation Create new training examples from old ones while preserving or correctly transforming their targets, so a model learns what matters instead of memo… Beginner12 min
  20. 18 Transfer learning and fine-tuning How to adapt a pretrained neural network to a new task when your labelled dataset is small. Beginner12 min
  21. 19 Model compression: pruning and distillation How to trade redundant weights, numerical precision, or teacher guidance for a model that actually fits and runs in production. Intermediate12 min
  22. 20 Mixed precision Use fp16 or bf16 for the expensive parts of training to reduce activation memory and speed up matrix math, while keeping numerically sensitive work… Advanced5 min
  23. 21 Activation checkpointing Understand what training VRAM contains, why activations grow with batch and sequence length, and how PyTorch and Transformers recompute selected fo… Advanced11 min
  24. 22 Multi-GPU: DDP & FSDP When the model fits on one GPU, replicate it (DDP). When it doesn't, shard it (FSDP). The memory math behind the decision, and how data parallelism… Advanced8 min
  25. 23 Pipeline parallelism When a model is too big to fit on one GPU, you split it across GPUs by layer — an assembly line of stages. The catch is the 'pipeline bubble' of id… Advanced9 min
  26. 24 Tensor parallelism and FSDP How FSDP shards training state and tensor parallelism splits computation so models larger than one GPU can train. Advanced12 min
  27. 25 Reinforcement learning foundations How agents learn by acting, receiving rewards, and improving decisions over time. Intermediate12 min
  28. Chapter 03

    Architectures

    6 lessons
  29. 26 Convolutional neural networks How CNNs see — convolution, kernels, feature maps, stride, padding, and pooling — built from scratch in NumPy, with an interactive filter you scan… Intermediate10 min
  30. 27 RNNs & LSTMs Before transformers, networks read text one word at a time, carrying a memory. How recurrent networks work, why their gradients vanish over long se… Intermediate8 min
  31. 28 Attention from scratch A mechanical, tensor-level explanation of attention: why it beats a fixed recurrent state, how Q, K, and V work, and where its quadratic cost bites. Intermediate12 min
  32. 29 Self-supervised learning How neural networks turn unlabelled data into useful representations through pretext tasks, contrastive learning, and masked prediction. Advanced12 min
  33. 30 Graph neural networks Graph neural networks let a model learn from entities and the relationships that connect them. Intermediate12 min
  34. 31 State space models and Mamba How state space models compress long histories into a small state, and how Mamba makes that memory selective. Advanced12 min
  35. Chapter 04

    Generative models

    8 lessons
  36. 32 GANs, VAEs & Diffusion Discriminative models tell cats from dogs; generative models can draw a new cat. Three families do it three different ways — and one of them now po… Advanced9 min
  37. 33 VAEs from scratch The generative-models overview said a VAE encodes to a latent and decodes, trained with reconstruction plus KL. This is the math underneath: maximi… Advanced8 min
  38. 34 GANs from scratch The overview said a GAN pits a generator against a discriminator. This is the math: the minimax objective, the alternating training loop, the non-s… Advanced8 min
  39. 35 Diffusion models (DDPM) The generative-models overview said diffusion adds noise forward and learns to reverse it. This is the math: the forward process has a closed form… Advanced8 min
  40. 36 Latent diffusion & Stable Diffusion Latent diffusion moves the expensive diffusion loop from pixels into a VAE's compressed latent space. This explains the Stable Diffusion pipeline,… Advanced8 min
  41. 37 Flow matching & rectified flow A practical explanation of flow matching: learn a velocity field that transports noise to data along a chosen probability path, then solve the resu… Advanced7 min
  42. 38 Evaluating generative models How do you score a generator when there is no single right answer? A practical guide to FID, KID, Inception Score, CLIP-based prompt adherence, cov… Intermediate7 min
  43. 39 Video, 3D & audio generation Diffusion, flow matching, and autoregression extend beyond images by changing the representation: time becomes video, scene geometry becomes 3D, an… Advanced7 min
  44. End of section 0 / 39 complete

    Make it stick — pass every quiz.

    Each lesson has a short quiz at the bottom. Passing the quiz is what marks the lesson complete and counts toward your certificate.

    Section complete 39 / 39 lessons

    Nice work — you finished Deep Learning.

    Certificates are earned per learning path, not per section. Here's where this section takes you:

FAQCommon questions

Deep Learning — frequently asked questions

Straight answers to the questions people ask most about deep learning.

What does PyTorch's autograd actually do?

Autograd records the operations in your forward pass as a graph, then automatically computes gradients of the loss with respect to every parameter by applying the chain rule backward. That's what lets you train with loss.backward() without deriving gradients by hand.

Why do neural networks need activation functions?

Without a non-linear activation, stacking layers just composes linear functions, which collapses to a single linear model no matter how deep. Activations like ReLU add non-linearity, letting the network approximate complex, curved decision boundaries.

What's the difference between SGD, Adam, and AdamW?

SGD updates weights using the raw gradient (optionally with momentum). Adam adapts the step size per parameter using running estimates of the gradient and its variance, often training faster. AdamW fixes how Adam handles weight decay for better regularisation, and is the common default for transformers.

What loss function should I use?

Use cross-entropy for classification — it heavily penalises confident wrong predictions — and mean squared error (or a robust variant like Huber) for regression. The loss must match the output layer, for example softmax outputs paired with cross-entropy.

What is a transformer and why did it change everything?

A transformer is an architecture built on self-attention, which lets every token directly weigh every other token regardless of distance, and processes a whole sequence in parallel rather than step by step like an RNN. That parallelism and long-range modeling are what made modern large language models possible.

Read the lesson