Small language models and on-device inference
How to choose, quantize, distil, evaluate, and run a small language model where a large one is wasteful.
What you'll learn
- Why most production LLM tasks do not need a frontier model
- How parameter count, quantization, KV cache, and memory bandwidth determine whether a model fits and feels fast
- Which capabilities degrade as models shrink, and which usually survive
- How a distilled specialist can beat a general-purpose small model
- How to evaluate and roll out a small-model swap without quietly damaging the product
Before you start
At 09:07 on Monday, ParcelCo’s support system receives a customer message:
“The box arrived crushed. Please refund the headphones. Order PC-48192.”
The production pipeline needs four things: classify the issue, extract the order number, detect urgency, and return valid JSON. It does not need a sonnet, a research plan, or a ten-step argument about refund policy.
Yet every message is sent to the company’s largest available model. The request takes 1.8 seconds, costs more than the task deserves, and sends a customer’s order number to a remote service. The model is being used as a very expensive regex with opinions.
This is the highest-leverage model decision in many LLM systems: use a smaller model when the job is narrow enough. A small language model is a model with relatively few parameters, often from a few hundred million to roughly 8 billion, chosen for a constrained task rather than broad general ability. On-device inference means running that model on the user’s phone, laptop, gateway, or an edge computer instead of calling a remote model API.
The point is not that small models are secretly as capable as large ones. They are not. The point is that capability is a curve, not a light switch. For a narrow task, the useful part of the curve may arrive long before the expensive part.
The capability-versus-cost curve
Model size buys capacity: more parameters can represent more patterns, languages, procedures, and exceptions. But the value of each additional parameter depends on the task. A 1B model may be adequate for assigning one of 12 support labels; a 7B model may handle messy extraction and multilingual messages better; a 70B model may be justified for an ambiguous policy question spanning several documents.
The costs rise with size:
- More weights occupy more memory.
- More bytes move through memory for each generated token.
- Hosted providers charge for more computation.
- Larger models load more slowly, use more energy, and make concurrency harder because each active request needs working memory.
The curve is uneven. A model may jump from unreliable to excellent after crossing a task-specific capacity threshold. Another four times as many parameters may then provide only a modest improvement. Parameter count is a budget estimate, not a quality guarantee; data, tokenizer, architecture, tuning, quantization, and task distribution matter too.
For ParcelCo, route routine extraction and classification to a small specialist, while sending ambiguous, multilingual, or policy-sensitive cases to a larger approved path:
This is a model routing problem as much as a model-size problem. Schema validation catches malformed output, not incorrect meaning. Rules catch only the risks they encode. For uncertainty, use a calibrated classifier, verifier, or selective-prediction score rather than raw model confidence. On a held-out set, measure error among cases kept on the small path against coverage before choosing a routing threshold.
First, make the memory arithmetic honest
If a model has P parameters and each uses b bits:
weight bytes = P × b / 8
For a 3-billion-parameter model:
| Representation | Bits per parameter | Raw weights |
|---|---|---|
| FP16 | 16 | 6.00 GB |
| INT8 | 8 | 3.00 GB |
| 4-bit | 4 | 1.50 GB |
| 3-bit | 3 | 1.125 GB |
These are decimal gigabytes and idealised weight sizes. Actual quantized files include scales, metadata, alignment, and sometimes mixed precision. A 7B 4-bit GGUF file is commonly around 4.3–4.8 GB, not exactly 3.5 GB. Quantization trades precision for a smaller memory footprint, and the lost precision may affect the task you care about.
The KV cache stores attention information from earlier tokens so the model need not recompute the whole conversation for every new token. It grows with context length and batch size; quantizing weights does not automatically shrink it.
A useful decoder estimate is:
KV bytes = 2 × layers × KV heads × head dimension × tokens × bytes per value
For 32 layers, 8 KV heads, head dimension 128, a 4,096-token context, and FP16 values:
2 × 32 × 8 × 128 × 4,096 × 2 = 536,870,912 bytes
That is 512 MiB for one sequence, or about 1 GiB at 8,192 tokens. The exact amount varies by architecture, cache precision, and runtime, but context has a real memory price.
Why on-device speed is mostly a bandwidth story
FLOPs measure arithmetic capacity. Memory bandwidth measures how quickly data moves between memory and processors. During decoding, the model generates one token at a time and must read much of its weights for each step. That makes bandwidth a frequent bottleneck.
If a 4.5 GB model runs on hardware with 80 GB/s of relevant bandwidth, moving the weights has an ideal lower bound of:
4.5 / 80 = 0.056 seconds
That is about 56 milliseconds per token, or 18 tokens per second, before arithmetic and runtime overhead. At 40 GB/s, the lower bound is about 112 milliseconds per token. A smaller model can therefore feel faster on the same chip because it moves fewer bytes. Prompt processing is more parallel and often more compute-heavy; generation is serial.
Operational measurements matter too. Phones throttle after sustained heat, cold starts include model loading and runtime setup, and each concurrent sequence needs its own KV cache. Benchmark warm, cold, and sustained workloads—not just the first response.
What shrinks badly, and what usually survives
The task’s structure matters more than the label “LLM”. Shrinking commonly hurts:
- Multi-step reasoning, which needs intermediate tracking and correction.
- Long-context recall and instruction-following with conflicting constraints.
- Rare-language coverage, especially low-resource languages and code-switching.
Bounded classification, extraction, routing, and short summaries often degrade less, particularly when the input is short and the schema is fixed. “Often” is not a promise: noisy text, rare languages, and too many fields can make even extraction fail. Test the actual distribution.
For ParcelCo, extracting order_id, issue_type, and urgency from a 450-token message is a safer small-model target than deciding an unusual refund exception from 30 pages of policy.
The specialist pattern: teach the small model your narrow job
A distilled specialist is trained to reproduce a stronger teacher’s useful decisions on one defined task. A practical process is:
- Collect representative tickets, including messy, incomplete, multilingual, and adversarial examples.
- Have a strong teacher create candidate labels or structured outputs, then review them.
- Fine-tune the small model on approved pairs, including the exact schema and refusal behaviour.
- Evaluate on a held-out set. Add schema validation and known-risk rules; route failures and measured high-risk cases to an approved larger fallback.
A general 3B model may know more facts than a distilled 1B router but perform worse on ParcelCo’s labels. The specialist spends its limited capacity on the company’s vocabulary, label boundaries, and JSON shape. See distillation and structured outputs.
The limitation is central: a specialist is narrow by design. Change the taxonomy, add a country, or introduce a product line and its accuracy can fall without any serving-stack error.
The decision is not always “small model”
| Choice | Use it when | Main advantage | Main risk |
|---|---|---|---|
| Large hosted model | Open-ended, rare, or high-consequence work | Broad capability | Cost, latency, and data exposure to the provider |
| General small hosted model | You need lower cost with remote fallback | Easy deployment | Network and provider costs remain |
| Distilled small model on device | Work is repetitive, private, and latency-sensitive | Local cost and predictable latency | Narrow coverage; you own updates and safety |
| Classical code or classifier | Rules and labels are stable | Tiny, cheap, testable | Breaks as ambiguity and exceptions multiply |
| Small-first cascade | Most requests are routine | Savings without abandoning hard-case quality | Requires routing and two systems |
Use code for a strict order-number pattern, a small model when wording varies but intent is bounded, and a larger model for broad knowledge or long reasoning. Use a cascade when small-model mistakes matter and difficult cases are uncommon.
On-device privacy applies only while data stays on the device. A hosted fallback may receive the order number or address precisely when the small model is uncertain. Keep fallback on-device or use an approved private service; otherwise redact or tokenize sensitive fields and obtain required consent. Define offline behaviour explicitly—queue, ask the user to retry, or return a safe review-needed result. Never silently use an unapproved provider.
On-device inference also means shipping model files, managing updates and licenses, handling hostile inputs, and testing across hardware.
Evaluation makes the swap safe
Compare models on the business contract, not on a pleasant demo. For ParcelCo, require approved issue_type labels, a pattern-matching or null order_id, one of three urgency values, review routing for missing evidence, and no invented refund decision.
Build a fixed, stratified evaluation set from production traffic: include languages, message lengths, spelling noise, missing fields, quoted messages, and prompt-injection attempts. Keep a private holdout. Measure field-level accuracy, schema validity, per-class precision and recall, language-specific errors, unsafe certainty, fallback rate, p95 and cold latency, sustained tokens per second, memory, temperature, and battery use.
Run the candidate in shadow mode, manually review disagreements, then canary a small percentage. Route schema failures, known-risk categories, unsupported languages, and cases above the held-out uncertainty threshold to the approved fallback. A malformed schema is not proof that the question is semantically difficult.
Failure modes: watch for the first symptom
- Rejected JSON: simplify the schema, use constrained decoding, and train on failed shapes; do not build heroic regex repair.
- Incomplete long-ticket summaries: improve context selection or chunking, then route genuinely global-recall tasks to a tested larger model.
- One language collapses: report metrics by language, add balanced data, or route unsupported languages upward.
- Fast demo, sluggish app: test sustained heat and memory pressure; reduce model size, generation length, or concurrency.
- High fallback rate: recalibrate the gate, simplify the task, or distil the specialist.
What to remember
- Choose size from the task’s error budget, not a leaderboard.
- Budget for weights, KV cache, runtime workspace, concurrency, and the operating system.
- Decode speed often follows memory bandwidth; heat and cold starts determine whether a benchmark survives on-device.
- Small models suit bounded tasks, while reasoning, long context, instruction conflicts, and rare languages need explicit testing.
- A distilled specialist with a measured fallback usually beats using either the largest or smallest model everywhere.
Quick check
Practice this in an interview
All questionsSmaller models win on latency, inference cost, on-device deployment, and fine-tuning feasibility. When trained on high-quality, curated data and aligned for a narrow task, a 7B–13B model can match or exceed a general-purpose 70B+ model on that specific workload while using a fraction of the compute budget.
Work top-down: start at the model layer with quantization, distillation, or routing cheaper models for easy requests, since model choices drive every downstream cost. Then optimize the runtime with batching, caching, and techniques like prompt caching for LLMs, and finally match infrastructure to the load using autoscaling on queue depth and spot or batch capacity. Track cost per token or per prediction alongside latency percentiles and accuracy so optimizations never silently degrade quality.
Model quantization represents weights and sometimes activations with fewer bits, reducing memory use and often improving inference cost or latency. More aggressive formats such as INT4 can reduce accuracy, but calibration, per-group scaling, outlier handling, and selective higher-precision layers can preserve quality; the result must be measured on the target workload and hardware.
Decide based on how fresh the prediction must be versus the cost and complexity of serving live. Use batch when results are needed every few hours or days, like daily churn lists, because it is cheap, simple, and can use spot or scheduled compute. Use real-time when a late or stale decision causes immediate loss, like fraud or ad auctions needing sub-100ms responses, accepting higher cost and complexity. Most production systems are hybrid: precompute heavy signals offline and do lightweight re-ranking online.