datarekha
Section 5 chapters · 44 of 44 lessons

NLP & Transformers

Natural language processing the way it actually evolved: classical text features and embeddings, the neural sequence models before attention, the transformer built from the ground up, the foundation models and LLMs that followed, and how transformers extended to vision and multimodal.

The NLP & Transformers journey 0 / 44 completed
  1. Chapter 01

    Classical NLP

    12 lessons
  2. 01 Text preprocessing Before any model sees language, text is a messy string. Classical NLP turns it into clean tokens through a pipeline — tokenize, normalize, remove s… Beginner7 min
  3. 02 Bag-of-words & TF-IDF Models need numbers, not words. The first and most durable answer is bag-of-words — represent a document as a vector of word counts — and TF-IDF, w… Beginner8 min
  4. 03 word2vec: CBOW & skip-gram TF-IDF has no idea that 'car' and 'automobile' mean the same thing. word2vec changed that — it learns dense vectors where similar words sit close,… Intermediate8 min
  5. 04 GloVe & fastText word2vec learns from local prediction windows and gives every word one vector. Its two influential successors each fix a different limitation: GloV… Intermediate8 min
  6. 05 Information retrieval & BM25 Bag-of-words gave you a representation; information retrieval is the system that uses it to find the relevant documents among millions, fast. Two p… Intermediate8 min
  7. 06 Topic modeling & LDA You have ten thousand documents and no labels — what are they about? Topic modeling discovers themes without supervision. LDA, the classic, rests o… Intermediate8 min
  8. 07 Sequence labeling: NER & POS Classification gives one label per document; many NLP tasks need a label per token — which words are names, which are verbs. That's sequence labeli… Intermediate8 min
  9. 08 Text classification: CNN & RNN Assigning a label to a whole text — sentiment, spam, topic — is the workhorse NLP task. Classical bag-of-words classifiers ignore word order; the n… Intermediate8 min
  10. 09 Language models before transformers Predicting the next word is the oldest task in NLP — and the exact objective GPT trains on. Before transformers it was n-gram counting and RNN lang… Intermediate8 min
  11. 10 Natural language inference Does the hypothesis follow from the premise? Natural language inference asks a model to label a sentence pair entailment, contradiction, or neutral… Intermediate7 min
  12. 11 Multilingual NLP Most NLP research is English, but there are thousands of languages. The key idea is a shared multilingual embedding space where translations land n… Intermediate7 min
  13. 12 Entity linking & knowledge graphs NER finds that 'Apple' is an entity, but which Apple — the company or the fruit? Entity linking disambiguates a mention to a specific knowledge-bas… Intermediate8 min
  14. Chapter 02

    Neural NLP (pre-transformer)

    6 lessons
  15. 13 Sequence-to-sequence models Translation, summarization, and question answering all map one sequence to another of different length. The seq2seq architecture solved this with a… Intermediate8 min
  16. 14 Attention (the RNN era) The seq2seq bottleneck forced the whole source through one fixed vector. Attention (Bahdanau, 2014) fixed it: at each decoding step, let the decode… Advanced8 min
  17. 15 Neural machine translation Seq2seq with attention is the translation engine — but translation as a task has its own machinery. How you decode (beam search beats greedy), how… Intermediate8 min
  18. 16 Text summarization Compressing a document to its essence splits into two very different approaches: extractive summarization selects the most important sentences from… Intermediate8 min
  19. 17 Question answering Answering a question from text comes in three flavors by where the answer lives and how it's produced. Extractive QA predicts a span of a passage;… Intermediate8 min
  20. 18 Dialogue systems Before ChatGPT, building a chatbot meant a pipeline. Task-oriented systems split into NLU, state tracking, policy, and generation; open-domain chat… Intermediate8 min
  21. Chapter 03

    Transformer architecture

    9 lessons
  22. 19 Tokenization & BPE A model never sees letters or words — it sees token IDs. How byte-pair encoding chops text into subwords, and why that explains cost, context limit… Beginner7 min
  23. 20 Self-attention The Q·K·V math that powers every modern LLM — derived step by step in NumPy with shape annotations at every line. Advanced10 min
  24. 21 Multi-head attention Run attention many times in parallel — each "head" learns a different relationship. The trick that makes transformers expressive. Advanced7 min
  25. 22 Positional encodings & RoPE Attention has no order — so we inject position. The original sinusoidal trick, and why modern LLMs all use RoPE. Advanced7 min
  26. 23 Inside the transformer block Attention is only half a transformer block. The feed-forward sublayer, residual connections, and normalization — plus why pre-norm and RMSNorm beca… Advanced9 min
  27. 24 The Transformer Architecture The full encoder-decoder Transformer from 'Attention Is All You Need', end to end — every block, why it exists, and how the same lego pieces become… Advanced11 min
  28. 25 FlashAttention Why attention is memory-bound, and how FlashAttention's tiling and online softmax compute the exact same result with far less GPU memory — making l… Advanced10 min
  29. 26 Sparse & sub-quadratic attention Beating O(n²) attention — sparse patterns, linear attention, and state-space models like Mamba, plus the hybrids (Jamba) that power today's long co… Advanced11 min
  30. 27 Differential attention Softmax attention never gives any token exactly zero weight, so it sprays a little attention on everything — 'attention noise' that buries the sign… Advanced8 min
  31. Chapter 04

    Foundation models & LLMs

    9 lessons
  32. 28 BERT, GPT, T5 Three transformer families with different pretraining objectives and different jobs. How to choose between encoder-only, decoder-only, and encoder-… Advanced8 min
  33. 29 Anatomy of a modern LLM What changed from the 2017 transformer to a modern LLM — RMSNorm, RoPE, grouped-query attention, and the SwiGLU gated FFN — with Qwen3 as the worke… Advanced11 min
  34. 30 Pretraining LLMs An LLM didn't learn facts from your prompt — it learned them from trillions of tokens of next-word prediction, before you ever arrived. How self-su… Intermediate8 min
  35. 31 Scaling laws Two LLMs, the same compute budget — 280B parameters or 70B. The smaller one wins. How test loss falls as a predictable power law in model size, dat… Advanced8 min
  36. 32 Multi-token prediction Standard language models predict one token at a time — myopic in training and wasteful at inference, where an expensive forward pass yields a singl… Advanced8 min
  37. 33 Mixture of Experts Frontier open models are huge on paper but cheap to run, because most of their parameters sit idle on any given token. How sparse Mixture-of-Expert… Advanced8 min
  38. 34 Frontier LLM walkthrough: DeepSeek-V3 You've met every modern part in isolation — RMSNorm, RoPE, MoE, MLA, MTP, FP8, DualPipe. A frontier model is the assembly of them into one cost-opt… Advanced10 min
  39. 35 Hugging Face transformers AutoTokenizer, AutoModel, pipeline — how to load any of 500,000 pretrained models with three lines of code and run inference. Intermediate8 min
  40. 36 LoRA & QLoRA fine-tuning Why fine-tuning a 7B model used to cost $50k — and how LoRA cut it to $50. The low-rank adapter trick that made fine-tuning practical. Advanced8 min
  41. Chapter 05

    Vision & multimodal

    8 lessons
  42. 37 Vision Transformers (ViT) CNNs ruled vision for a decade by baking in convolution's inductive biases. The Vision Transformer asked the heretical question: what if you just r… Advanced7 min
  43. 38 CLIP: contrastive vision-language A ViT encodes images, a transformer encodes text — but in separate spaces you can't compare. CLIP trains both encoders together with a contrastive… Advanced8 min
  44. 39 Vision-language model architectures CLIP aligns images and text; a vision-language model lets an LLM actually see — answer questions about an image, read a chart, reason over a screen… Advanced8 min
  45. 40 VLM model families There are dozens of vision-language models — Qwen-VL, InternVL, Pixtral, and more. Rather than catalogue brands, learn the axes that actually disti… Advanced7 min
  46. 41 Unified multimodal models A vision-language model can see but not draw — it understands images and generates only text. Unified multimodal models do both: one model that per… Advanced7 min
  47. 42 Omni & embodied models The multimodal recipe — turn everything into tokens, predict the next one — doesn't stop at images. Extend it to audio and you get omni models: rea… Advanced7 min
  48. 43 Video & document understanding A VLM reads one image, but two domains push the limits: video, which is too many frames, and dense documents, which are too much detail. Both reduc… Advanced7 min
  49. 44 Multimodal RAG Standard RAG retrieves text, but knowledge lives in charts, scanned PDFs, and slides — and the OCR-then-parse pipeline flattens all of it. Multimod… Advanced7 min
  50. End of section 0 / 44 complete

    Make it stick — pass every quiz.

    Each lesson has a short quiz at the bottom. Passing the quiz is what marks the lesson complete and counts toward your certificate.

    Section complete 44 / 44 lessons

    Nice work — you finished NLP & Transformers.

    Certificates are earned per learning path, not per section. Here's where this section takes you:

Skip to content