Fine-tuning: LoRA, QLoRA & PEFT

In the previous lesson you decided the honest thing: the model already has the facts, but it won’t behave the way you need — wrong format, wrong tone, a missing skill. That is the one case where retrieval can’t help and fine-tuning genuinely earns its place. So you roll up your sleeves to fine-tune — and walk straight into a wall.

Why you can’t just retrain everything

A 7-billion-parameter model has 7 billion weights. “Fine-tune it” sounds like nudging a few of them. It isn’t. Full fine-tuning computes a gradient for every weight, and the Adam optimizer keeps two extra running numbers per weight, so the memory bill is several times the model’s own size before a single batch runs. Worse, the output is a brand-new complete copy of the model.

Put numbers on it. One 7B checkpoint in 16-bit precision is about 14 GB on disk. Train it five different ways — support, sales, legal, code, German — and you are storing five 14 GB files, 70 GB of almost-identical weights that differ in a tiny fraction of their values. For a 65B model, the full-fine-tune memory alone overflows a single GPU before training even starts. There has to be a cheaper way to say “behave a little differently.”

The idea: freeze the book, add margin notes

Picture a 1,000-page textbook you mostly agree with. To fix one chapter you don’t reprint the book — you slip in a thin stack of transparent overlays with your corrections, and the reader sums each overlay with the page beneath it. The book stays frozen; your edits are small, cheap, and swappable.

That is exactly the move. Freeze the original weights. Learn a small, separate update that adds on top. This family of techniques has a name: PEFT — Parameter-Efficient Fine-Tuning — and its most-used member is LoRA.

LoRA, precisely

LoRA stands for Low-Rank Adaptation. Here is the whole idea in one sentence: instead of learning a full-size change to a weight matrix, learn a low-rank one — and a low-rank matrix factors into two skinny matrices, so it has far fewer numbers.

A linear layer holds a weight matrix W of shape d × d. Full fine-tuning learns a change ΔW of the same d × d shape — just as many numbers as W itself. LoRA bets that the useful ΔW is low-rank, meaning it can be written as a product of two thin matrices, B (shape d × r) and A (shape r × d), where the rank r is small — 8, maybe 16. You freeze W and train only A and B.

Trace the rank, one matrix at a time

Take a single attention projection in a 7B-class model — W is 4096 × 4096.

• Full fine-tuning updates every one of 4096 × 4096 = 16,777,216 numbers in this one matrix.
• LoRA at rank 8 trains B (4096 × 8) and A (8 × 4096): 4096×8 + 8×4096 = 65,536 numbers.
• That is 65,536 / 16,777,216 = 0.391% of the matrix — about one weight in 256.

The rank is the dial. Turn it and the count moves with it:

Low rank means few parameters but limited capacity to change behavior; higher rank buys expressiveness at the cost of more to train and store. Rank 8–16 is the common starting point. At inference you compute h = W·x + (B·A)·x — the frozen path plus the small learned overlay — and you can even merge B·A back into W afterward, so there is zero extra latency at serving time.

The payoff is the storage story flipping on its head: you keep one frozen base model plus a stack of adapters — the trained A and B for each task, a few megabytes each. Five behaviors is one 14 GB base + five small files, not five 14 GB models.

QLoRA — when even the frozen base won’t fit

LoRA shrinks what you train, but the frozen base still has to sit in GPU memory while you train. For a 65B model in 16-bit that base is 65e9 × 2 bytes = 130 GB — three 48 GB GPUs before you have trained anything.

QLoRA adds one move: quantize the frozen base to 4-bit. The base is read-only during training, so the precision it loses matters far less than it would for a model you were still updating. At 4-bit the same 65B base is 65e9 × 0.5 bytes = 32.5 GB — now it fits on a single 48 GB GPU with room left for the activations and the small adapters. The adapters themselves stay in higher precision, because they are the part actually learning. The original QLoRA work used exactly this trick to fine-tune a 65B model on one 48 GB GPU.

The answer is the whole point of the technique: you keep the single frozen base resident and swap the matching few-MB adapter per request. That is only possible because the base never changed — every adapter was trained against the same frozen weights, so they are interchangeable overlays on one shared book.

In one breath

• Full fine-tuning is impractical at scale: it trains every weight, carries a multiplied optimizer-memory bill, and saves a full model copy per task.
• PEFT is the family of parameter-efficient methods; LoRA is the workhorse — freeze W, learn a low-rank update ΔW = B·A, train only the skinny A and B.
• The rank r is the dial: at r = 8 one 4096×4096 matrix trains 65,536 params — 0.391% of it — trading capacity for cost.
• You store one frozen base plus few-MB adapters, and can merge B·A into W for zero extra inference latency.
• QLoRA quantizes the frozen base to 4-bit (130 GB → 32.5 GB for 65B), so big models fine-tune on a single GPU; adapters stay higher-precision because they are the part that learns. LoRA changes behavior, not fresh facts.

Quick check

Next

Fine-tuning adapts a model to your behavior. But how does a raw pretrained model learn to follow instructions and be helpful in the first place? That is alignment — SFT, RLHF & DPO.