LoRA and QLoRA fine-tuning
Why fine-tuning a 7B model used to cost $50k — and how LoRA cut it to $50. The low-rank adapter trick that made fine-tuning practical.
What you'll learn
- The math behind LoRA's low-rank decomposition
- How QLoRA combines 4-bit quantization with LoRA
- When to reach for plain LoRA vs QLoRA — the memory/speed trade-off
- When to fine-tune vs just prompt engineer
Before you start
Fine-tuning a 7B model the old-fashioned way means updating 7 billion parameters — needing optimizer state for each (often 4x in fp32), plus gradients, plus activations. Full fine-tuning a 7B model fits on a single A100 only if you’re careful, and the resulting checkpoint is the same size as the original. Now imagine doing this for 50 different domains.
LoRA changed this. Instead of updating the weight matrices directly, train tiny adapter matrices alongside them (an adapter is a small add-on module that sits beside a layer without changing its weights). The original model stays frozen (its weights are locked and never updated) — you only train and ship the adapters, which are typically less than 1% of the parameters.
Drag the rank to see how few parameters LoRA trains
W₀ is frozen. LoRA learns two thin matrices A (r×d) and B (d×r) whose product B·A is the low-rank update added to W₀. Raise rank r to see the update grow richer — and watch the parameter count stay linear in r, not quadratic.
The math
For a frozen weight matrix W ∈ ℝ^(d × k), LoRA learns two small
matrices:
A ∈ ℝ^(d × r) (small)
B ∈ ℝ^(r × k) (small)where r is much smaller than d or k — typically r = 8, 16,
or 64. The model uses W + α·A·B instead of W. Only A and B
are trained; W is frozen.
The product A·B has shape (d, k) — same as W — but its rank
(the number of independent directions it can represent) is at most r.
Because r is tiny, this is called a low-rank update. The bet:
useful fine-tuning changes lie in a low-rank subspace, so a small A·B
captures them with far fewer parameters than rewriting all of W.
r, so the trainable count collapses from d·k to (d+k)·r.import numpy as np
# Imagine a frozen weight: d=1024, k=1024 -> 1,048,576 params
d, k = 1024, 1024
W = np.random.randn(d, k).astype(np.float32) * 0.01
# Full fine-tuning would train d*k = 1,048,576 params
# LoRA with rank r=16 trains d*r + r*k = 1024*16 + 16*1024 = 32,768 params
# That's 32x fewer!
r = 16
A = np.random.randn(d, r).astype(np.float32) * 0.01
B = np.zeros((r, k), dtype=np.float32) # init B = 0 so A@B = 0 at start
total_full = d * k
total_lora = d * r + r * k
print(f"Full fine-tune params: {total_full:,}")
print(f"LoRA params (r={r}): {total_lora:,}")
print(f"Reduction: {total_full / total_lora:.1f}x")
# During inference, you can either keep them separate (W + A@B)
# or merge them: W_merged = W + alpha * A @ B (one-time cost, no runtime overhead)
alpha = 16
delta = alpha * (A @ B) / r # alpha/r scaling is conventional
print(f"\nDelta shape (same as W): {delta.shape}")
print(f"Delta rank: at most r = {r}")Full fine-tune params: 1,048,576
LoRA params (r=16): 32,768
Reduction: 32.0x
Delta shape (same as W): (1024, 1024)
Delta rank: at most r = 16The shapes and counts are what matter: the update A·B is the full (1024, 1024)
size of W, yet it is built from only 32,768 trainable numbers — a 32×
saving at r = 16, and the saving grows as the matrices get bigger.
For a 7B model with LoRA at r=16 applied to all attention projections,
you’re typically training 20–40 million parameters — 0.3% of the
total. Optimizer state, gradients, and checkpoints all shrink
proportionally.
QLoRA — even more squeeze
QLoRA combines LoRA with 4-bit quantization of the base model.
The frozen W is stored in 4-bit (NF4 format), saving 8x memory vs
fp32. LoRA adapters stay in bf16 — they’re tiny anyway.
Result: fine-tune a 65B model on a single 48GB GPU. The 2023 QLoRA paper made open-source fine-tuning real.
| Approach | 7B model memory (training) | 65B model memory (training) |
|---|---|---|
| Full fine-tune (fp32) | ~120 GB | impossible on single GPU |
| Full fine-tune (bf16) | ~60 GB | ~520 GB |
| LoRA (bf16 base) | ~16 GB | ~140 GB |
| QLoRA (4-bit base) | ~6 GB | ~48 GB |
LoRA vs QLoRA — which should you reach for?
Both train the same tiny adapters. The only thing that changes is how the frozen base model is stored while you train — and that one choice decides whether the job fits on your GPU, at the cost of a little speed when it does.
| LoRA | QLoRA | |
|---|---|---|
| Base stored as | bf16 — 2 bytes/param | 4-bit NF4 — 0.5 bytes/param |
| 7B training VRAM | ~16 GB | ~6 GB |
| 65B training VRAM | ~140 GB (2× A100) | ~48 GB (one A100) |
| Training speed | baseline | ~30% slower — weights de-quantized every forward pass |
| Final quality | baseline | within ~1 point on most benchmarks |
| Adapters produced | identical | identical |
The mental model: QLoRA buys memory with time. Squeezing the frozen weights to 4 bits frees the VRAM to fit a bigger model — but the GPU must de-quantize them back to bf16 on every step, so each step runs a little slower.
So the rule is simple:
- Fits in bf16? Use plain LoRA — it’s faster and marginally higher quality. If your 7B fine-tune sits comfortably on a 24 GB card, there’s no reason to quantize the base.
- Memory-bound? Use QLoRA — when bf16 would overflow the GPU (a 13B on a 16 GB card, a 70B on a single A100), 4-bit is what lets the job start at all. A slower run beats one that can’t begin.
In code
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
# 4-bit quantization for the base model
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
quantization_config=bnb_config,
device_map="auto",
)
# LoRA config
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 13M | total params: 8B | trainable%: 0.17%lora_alpha is the scaling factor; the effective update is
(alpha/r) * A @ B. A common choice is alpha = 2 * r.
target_modules is the list of layer names where LoRA adapters get
attached. Conventional choice: all four attention projections
(q_proj, k_proj, v_proj, o_proj). More aggressive: also include
MLP layers (gate_proj, up_proj, down_proj).
After training, you save just the adapters:
model.save_pretrained("./lora-adapter") # tens of MB, not GBAt inference time you load the base model and apply the adapter:
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
model = PeftModel.from_pretrained(base, "./lora-adapter")Or merge for deployment:
merged = model.merge_and_unload()
# `merged` is now a normal HF model — A@B has been added to WLoRA, QLoRA, and the future
Variants keep appearing — DoRA, VeRA, AdaLoRA — each squeezing a bit more quality, but the core idea is unchanged. The PEFT library implements them all behind a unified interface.
For production today: QLoRA is the default fine-tuning recipe unless you specifically need full fine-tuning.
In one breath
- Full fine-tuning updates every weight (plus optimizer state and a full-size checkpoint per domain) — expensive and unwieldy.
- LoRA freezes
Wand learns a low-rank updateα·A·B(withA: d×r,B: r×k,rtiny), betting useful fine-tuning changes live in a low-rank subspace — so you train under 1% of the parameters and ship adapters of tens of MB. - QLoRA stores the frozen base in 4-bit NF4 while adapters stay bf16 — it buys memory with a little time (de-quantize each step), fitting a 65B fine-tune on a single 48 GB GPU. Rule: fits in bf16 → plain LoRA; memory-bound → QLoRA.
- Adapters merge back into the base (
merge_and_unload()) for zero-overhead inference; and fine-tune at all only when prompting + RAG plateau — its strongest case is cutting inference cost by matching a frontier model on your distribution with a small open one.
Quick check
Quick check
Practice this in an interview
All questionsLoRA (Low-Rank Adaptation) freezes the original model weights and injects trainable low-rank decomposition matrices into attention layers. This cuts the number of trainable parameters by 100x-1000x while matching or approaching full fine-tuning quality, making it practical on a single GPU.
Catastrophic forgetting is when fine-tuning on a new task overwrites weights and erases previously learned capabilities. Parameter-efficient methods like LoRA freeze the base weights and train only small added parameters, preserving the original knowledge while adapting behavior, and techniques like lower learning rates, replay data, and adapter isolation further reduce forgetting.
QLoRA quantizes the frozen base model to 4-bit using NF4 and double quantization, then trains LoRA adapters on top, with paged optimizers to handle memory spikes. This dramatically lowers the memory footprint, enabling fine-tuning of large models on a single consumer GPU with little quality loss versus 16-bit LoRA.
Prompt engineering is the right starting point when the task can be described in natural language, the required knowledge already exists in the base model, and iteration speed matters — no training required. Fine-tuning is warranted when you need consistent output format at scale, domain-specific style that prompts cannot reliably impose, or when latency and token costs from long system prompts are prohibitive.
LoRA freezes the pretrained weights and injects small trainable low-rank matrices into selected layers, learning the weight update as their low-rank product. This trains a tiny fraction of parameters, slashing memory and storage while approximating full fine-tuning, and the adapters can be merged back at inference.
Smaller models win on latency, inference cost, on-device deployment, and fine-tuning feasibility. When trained on high-quality, curated data and aligned for a narrow task, a 7B–13B model can match or exceed a general-purpose 70B+ model on that specific workload while using a fraction of the compute budget.