What is LoRA and how does it make fine-tuning parameter-efficient?
LoRA is a parameter-efficient fine-tuning method that freezes pretrained weights and learns a low-rank update through two small trainable matrices. Because those matrices contain far fewer parameters than the original weights, LoRA reduces optimizer memory and adapter storage, and the update can be merged into the base model for inference.
How to think about it
LoRA, short for Low-Rank Adaptation, is a parameter-efficient fine-tuning method: it freezes the pretrained weight matrices and trains two much smaller matrices whose product is the task-specific update. Because only those low-rank adapters receive gradients and optimizer state, training and storing a task version can be dramatically cheaper, while the adapter can be merged into the base weights for deployment.
Why the saving happens
An interviewer is probing whether you understand where the saving comes from, not whether you can expand the acronym.
In full fine-tuning, a layer’s weight matrix W is updated directly. If W has d_out rows and d_in columns, the update contains d_out × d_in trainable values. That is expensive for a large language model because every trainable value usually needs a gradient and optimizer state. Standard Adam, for example, keeps two moment estimates for each trainable parameter.
LoRA keeps W frozen and represents the update as:
W' = W + ΔW
ΔW = (α / r)BA
Here, A has shape r × d_in, B has shape d_out × r, and r is the rank, meaning the size of the narrow bottleneck between them. α is a scaling factor. The trainable parameter count becomes:
r × (d_in + d_out)
When r is much smaller than either dimension, that is far below d_out × d_in.
The mechanism is easy to picture. A compresses the input into r learned directions. B expands those directions back into the layer’s output space. The model is not allowed to learn an arbitrary change to the layer; it learns a change assembled from a small number of directions. LoRA works when the useful change for the new task is approximately low-dimensional.
Common misconception: LoRA does not make the pretrained matrix W low rank, and it does not replace W with BA. The original matrix stays intact. Only the correction ΔW is constrained to have low rank. A full-rank base model plus a low-rank correction can still produce rich behavior.
A common initialization makes A random and B zero. That means BA is initially zero, so the adapted model starts with exactly the same output as the pretrained model. After the first updates, the adapter begins changing the result.
A concrete calculation
Suppose one transformer projection is a 4096 × 4096 matrix. Full fine-tuning gives it 16,777,216 trainable parameters. With LoRA rank r = 16, the adapter has:
16 × (4096 + 4096) = 131,072 parameters
That is 128 times fewer trainable parameters for this matrix.
Now suppose the model has 32 blocks and we attach LoRA to the query and value projection in each block. That is 64 matrices:
| Scope | Full fine-tuning | LoRA with r = 16 |
|---|---|---|
One 4096 × 4096 matrix | 16,777,216 | 131,072 |
| 64 query/value matrices | 1,073,741,824 | 8,388,608 |
| Raw fp16 tensor data | About 2 GiB | About 16 MiB |
The last row is only the raw trainable tensor data. It excludes the frozen base model, activations, metadata, and optimizer details. The important point is not that LoRA removes the base model. It does not. The saving comes from avoiding gradients and optimizer states for roughly a billion selected weight values, and from saving a small adapter rather than a complete task-specific model copy.
What the production pattern looks like
First, choose which modules receive LoRA. Classic recipes target the query and value projection layers in attention. Other recipes target query, key, value, and output projections, or also the feed-forward projections. There is no universal best choice. A narrow target is cheaper, but it may not give the model enough freedom for a large domain shift. Module names also differ between model architectures, so inspect the actual model rather than copying a target list blindly.
Next, choose the rank and scaling. Rank controls capacity: a higher rank can represent a broader update but uses more memory and storage. Values such as 8, 16, or 32 are reasonable starting experiments, not laws of nature. The scaling factor controls the size of the update through α / r; it does not add expressive capacity. Increasing α cannot compensate for an adapter whose rank is too small.
During training, the base weights remain frozen and the optimizer receives only the adapter parameters, plus any explicitly selected extra parameters such as an output head or embeddings. Count the trainable parameters before starting. If the count is close to the whole model, the setup is not parameter-efficient. If it is zero, the model will produce a very convincing training log while learning absolutely nothing.
The saved adapter should include its configuration: rank, scaling, target modules, dropout if used, and the exact base-model identity. The adapter is not normally a standalone model. It is a small patch that must be loaded on the compatible base model.
For one fixed task, the adapter can be merged:
W_merged = W + (α / r)BA
After merging, the serving path can use an ordinary linear layer, with no separate LoRA branch. If one base model serves many tasks, keeping adapters separate is often better because they can be swapped without storing a full model for every task. Stacking or combining several adapters is not automatically equivalent to training them jointly; test that behavior rather than assuming the patches will cooperate.
The nuance that separates a good answer
“Parameter-efficient” does not mean “free” or even “faster in every way.” The forward pass still uses the frozen base weights. Long sequences and large batches still require activation memory. An unmerged adapter also adds a small extra computation path. LoRA primarily reduces trainable-parameter memory, optimizer memory, and checkpoint size.
It also does not solve the base-model memory problem by itself. A 70-billion-parameter model in fp16 needs roughly 140 GB of raw weight storage before runtime overhead. LoRA alone will not put that model on a 12 GB GPU. QLoRA is a related approach that combines LoRA with quantization of the frozen base model. Quantization is a separate memory-saving technique; LoRA itself does not quantize anything.
LoRA is a weaker choice when the new task requires broad changes across the model, changes to vocabulary or embeddings, or maximum possible adaptation quality. You can attach adapters to more modules, train selected embeddings, increase the rank, or use full fine-tuning. Each option gives up some of LoRA’s simplicity or savings. The right comparison is validation quality at a fixed memory and serving budget, not a blanket claim that LoRA is always better.
A failure mode worth naming
A common production symptom is that the training loss stays almost identical to the frozen base model’s loss, or validation quality never moves, even though the job reports successful steps.
Check the number of trainable parameters and the gradient norms of A and B after a backward pass. The optimizer may have been created before the adapters were added, or a later freezing step may have set every parameter to non-trainable. Zero-initialized B explains why the first forward pass matches the base model; it does not explain an unchanged model after many optimizer steps.
The opposite symptom is an out-of-memory error despite a tiny adapter checkpoint. In that case, the frozen model weights or activations are probably the problem. LoRA cannot reduce memory consumed by sequence length, attention activations, or the raw base model. Quantization, gradient checkpointing, shorter sequences, or a smaller batch may be needed separately.
What they’ll ask next
How does LoRA compare with full fine-tuning?
Full fine-tuning updates every model weight, so it has maximum flexibility but requires more gradient memory, optimizer state, and storage. LoRA updates a constrained low-rank correction, making multiple task-specific adapters cheap to train and swap. Full fine-tuning can win when the task requires a broad change, but LoRA can match it closely on many behavior and domain-adaptation tasks.
What do rank and alpha mean, and how would you choose them?
Rank is the adapter’s capacity: it determines how many independent directions the update can represent. Alpha scales the update through α / r; it does not increase capacity. I would start with a small rank, compare held-out task metrics, and increase rank if the adapter is clearly underfitting. I would tune alpha separately rather than using it as a substitute for rank.
Is LoRA the same as QLoRA, and does LoRA reduce inference latency?
No. LoRA learns low-rank adapters. QLoRA additionally stores or runs the frozen base model using quantization. An unmerged LoRA adapter adds a small inference path, while a compatible merged adapter folds BA into the original weight and removes that extra path. Whether merging is available and numerically safe depends on the serving and quantization stack.
Say this in the interview
“LoRA freezes the expensive pretrained weights and learns a small low-rank correction, trading some adaptation capacity for far lower trainable-state and storage cost; quantization is a separate optimization, as in QLoRA.”