You have a 70B model, limited GPU memory, and a few hundred thousand high-quality examples. How would you choose between full fine-tuning, LoRA, QLoRA, and another PEFT method, and how would you detect that the adapter is memorizing or degrading general capabilities?
Start with QLoRA because it keeps the 70B base frozen and quantized, making training feasible on limited memory, then compare against bf16 LoRA if memory and quality justify it. Use full fine-tuning only when low-rank updates demonstrably underfit a broad change, and detect memorization or regression with source-aware held-out data, extraction probes, and paired evaluations against the original base model.
How to think about it
With a 70B model and limited GPU memory, I would start with QLoRA: keep the 70B base frozen in 4-bit storage and train a small LoRA adapter, then compare it with ordinary LoRA if memory allows. I would reserve full fine-tuning for a demonstrated need for broad, high-capacity change, and use DoRA as the next experiment when ordinary LoRA underfits; I would detect memorization and capability loss with source-aware held-out data, extraction probes, and paired pre/post regression tests against the base model.
Why the memory changes the answer
Full fine-tuning updates every model weight. A 70B model has 70 billion of them. In one common mixed-precision setup, the rough memory bill looks like this:
| Item | Approximate memory |
|---|---|
| BF16 weights | 140 GB |
| BF16 gradients | 140 GB |
| FP32 Adam moments | 560 GB |
| FP32 master weights | 280 GB |
| Total before activations | 1.12 TB |
Those are decimal gigabytes, and the exact total depends on the optimizer and sharding strategy. Activations, temporary buffers, and communication add more. FSDP or ZeRO can divide this burden across GPUs; they do not make the total state disappear.
LoRA, or Low-Rank Adaptation, freezes the base model and learns a low-rank update to selected weight matrices. Instead of learning every entry in a matrix W, it learns ΔW = B A, where A and B are much smaller matrices. For an illustrative 8192 × 8192 layer at rank r=16, the full matrix has 67,108,864 parameters, while the LoRA update has 16 × (8192 + 8192) = 262,144. That is 256 times fewer parameters for that layer.
The base still consumes memory, but gradients and optimizer states are needed only for the adapter. That makes ordinary LoRA attractive when the BF16 base can fit through GPU sharding or offload.
QLoRA is not a different kind of update. It is LoRA with the frozen base stored in a low-bit format, usually 4-bit NormalFloat storage, while computation typically uses BF16 or FP16. The raw storage calculation for 70B parameters is 70 billion × 4 / 8 = 35 GB. Quantization scales and metadata make the real footprint larger, and activations still matter, especially at long sequence lengths. A 70B QLoRA run may therefore be possible on a 48 GB GPU only with careful sequence length, microbatching, gradient checkpointing, and runtime settings. An 80 GB GPU or multiple GPUs gives considerably more breathing room.
The important distinction is this: QLoRA saves memory by compressing the frozen base, not by making the trainable update more expressive. Its quality ceiling is still affected by the base model, the quantization noise, the adapter rank, and which modules receive LoRA updates.
How I would choose
I would make the choice experimentally, but not by launching four expensive runs blindly.
- QLoRA first when GPU memory is the binding constraint and the task is supervised adaptation: support behavior, a domain-specific assistant, structured output, or an instruction style.
- LoRA next when the quantized run loses quality on exact formatting, subtle reasoning, or a sensitive domain, and the BF16 base can fit with sharding.
- DoRA when LoRA appears to underfit at a reasonable rank. DoRA, or Weight-Decomposed Low-Rank Adaptation, separates a weight’s magnitude from its direction. It can give the update more freedom than a plain low-rank direction alone, but adds implementation and serving complexity.
- Full fine-tuning only when the desired change is broad enough that a low-rank update cannot represent it, and the team can afford distributed training, checkpoint storage, careful rollback, and a serious regression program.
Another compact option is IA3, which learns vectors that rescale intermediate activations rather than learning matrix updates. It is extremely parameter-efficient, but that constraint can be too severe for a substantial behavioral or knowledge shift. Smaller is not automatically wiser; sometimes it is simply less able to say what you asked.
The number of examples does not settle the decision. Two hundred thousand near-duplicates are not two hundred thousand independent lessons. I would measure diversity, sequence length, source distribution, and how far the examples are from the base model’s existing behavior.
A concrete run
Suppose the model is a customer-support assistant for a medical-device company. The team has 250,000 carefully reviewed conversations, policy answers, and structured troubleshooting examples.
I would first deduplicate exact matches and near-duplicates, then split by document, customer account, and time period rather than randomly splitting individual turns. For example:
- 220,000 examples for training
- 15,000 examples for validation and early stopping
- 15,000 examples held untouched for the final test
A random split could place nine paraphrases of the same policy paragraph in both training and test. The resulting score would be precise, repeatable, and mostly meaningless.
I would run QLoRA with a small rank sweep such as r=8, r=16, and r=32. I would start by targeting the attention projections and the model’s feed-forward linear projections, using the module names appropriate to that architecture rather than assuming every 70B model uses the same names. I would monitor validation loss and task metrics after each evaluation interval, not merely after one epoch.
Suppose the r=16 adapter raises policy-answer accuracy from 71 percent with the base model to 84 percent on the held-out test set. That sounds good. Now suppose its training loss falls from 1.9 to 0.35, while validation loss bottoms out at 0.68 and later rises to 0.91. The rising validation loss is an early warning that the adapter is learning the training set more specifically than the underlying task.
At that point I would try earlier stopping, fewer epochs, a lower learning rate, less aggressive module targeting, or a smaller rank. I would also mix in a measured sample of general instruction data or add a loss that keeps the adapted model close to the base model on generic prompts. The right fix depends on whether the problem is memorization, an overly broad update, or a genuinely difficult domain shift.
Detecting memorization and general degradation
I would evaluate three separate things.
First, task generalization. The test set must be source-aware and untouched. Report results by slice: product family, language, customer type, policy age, prompt length, and rare cases. A single average can hide the model failing exactly where the 3 a.m. incident will occur.
Second, memorization. I would search for:
- Verbatim reproduction of training conversations, including irrelevant surrounding text.
- Exact continuation of unique strings, internal case numbers, or planted canaries.
- High confidence on training examples but weak performance on paraphrases and new entities.
- A large gap between train loss and source-separated validation loss.
- Outputs that reveal personal, confidential, or licensed text.
For controlled testing, I can place unique harmless canary strings in a known training subset and test whether the adapted model reproduces them from partial prompts. I would compare both likelihood and generated text against the original base. A canary that the base does not produce but the adapter emits reliably is strong evidence that the update stored specific training content. This is not a perfect privacy audit, but it is far better than staring at the training loss and hoping.
Third, capability regression. I would freeze a regression set before training: perhaps 1,000 prompts covering general instruction following, reasoning, coding, factual questions, multilingual behavior, refusal boundaries, and any capabilities the product depends on. Run the base and adapted models with the same tokenizer, prompt template, decoding settings, and evaluator. Compare paired results, not unrelated benchmark runs.
If the domain score rises from 71 to 84 percent but the general regression set falls from 78 to 76 percent, I would not call the run finished. I would inspect slices and confidence intervals, then decide whether two percentage points are noise or a real product cost.
The common misconception is that LoRA cannot cause forgetting because the base weights are frozen. It is true that the original checkpoint remains unchanged. But the served model uses the effective weights W + ΔW; a sufficiently strong adapter can shift logits across many unrelated prompts. Catastrophic forgetting can therefore appear as changed behavior even when the base is safely stored on disk.
The senior-level trade-off
Full fine-tuning offers the most freedom, but freedom makes unwanted change easier. LoRA and QLoRA are cheaper, easier to version, and allow one base model to serve many adapters, but their low-rank bottleneck can underfit a large domain transformation. DoRA may recover some quality at low rank, but it is not free: it brings extra parameters, compute, and serving compatibility questions.
I would choose the smallest method that clears the task metric without failing the regression and privacy gates. If no adapter can meet the requirement, that is evidence for a larger update, not an excuse to increase the rank forever.
What they’ll ask next
How would you choose the LoRA rank?
I would sweep a few ranks, such as 8, 16, and 32, while holding the data and evaluation fixed. Rank controls the update’s capacity, not its quality by itself. I would choose the smallest rank that reaches the target without worsening validation loss or regression slices.
Does QLoRA produce a 4-bit model at inference?
Not necessarily. QLoRA stores the frozen base in 4-bit form during training and learns the adapter separately. The serving representation depends on the inference runtime: it may keep the base quantized and apply the adapter, or use another supported representation. I would verify the runtime’s current adapter and quantization support before promising a deployment shape.
What would you do after finding regression?
First check the split and prompt template. Then try early stopping, lower learning rate, less broad module targeting, lower rank, and a small replay mixture of general examples. I would also test a base-model-preservation or distillation loss on generic prompts. If the task still underfits, I would compare DoRA or full fine-tuning rather than hiding the regression behind a better-looking task average.
One line to say in the room
“I’d start with QLoRA for the memory budget, prove its value against BF16 LoRA where possible, and treat source-aware generalization, extraction probes, and base-versus-adapter regression as release gates rather than afterthoughts.”