What is model quantization, and how does it affect quality?

How to think about it

When an 8-billion-parameter model barely fits on a GPU, model quantization represents its weights, and sometimes its activations, with fewer bits so it uses less memory and moves data faster. The price is approximation: rounding and clipping introduce error, so quality can fall as precision becomes more aggressive, especially with poorly calibrated INT4 quantization.

Why quantization matters

At inference time, a transformer repeatedly multiplies weights by activations, which are the intermediate values produced as tokens pass through the network. Those weights must be stored and moved through memory. With batch-one serving, the bottleneck is often moving data rather than doing arithmetic. Halving the bytes per weight can therefore reduce memory pressure and sometimes latency.

For 8 billion parameters, the raw weight memory looks like this:

Format	Bits per value	Raw weight memory
FP32	32	32.0 GB
FP16 or BF16	16	16.0 GB
INT8	8	8.0 GB
INT4	4	4.0 GB

Those figures use decimal gigabytes and ignore scales, zero-points, metadata, runtime buffers, and the attention key-value cache. An 8B model in FP16 therefore needs 16 GB just for weights. On a 24 GB GPU, that leaves only about 8 GB for everything else. An INT4 version may need roughly 4 GB for the quantized weights plus overhead, leaving considerably more room for longer context or a larger batch.

Quantization does not remove parameters. It stores each parameter more compactly.

How the conversion works

A floating-point weight can take many values. A quantized weight uses one of a much smaller set of integer codes. The quantizer records a scale, which tells the runtime how large each code step is, and sometimes a zero-point, which shifts the integer range.

A simplified affine quantizer is:

q = round(x / s) + z

and reconstruction is:

x_hat = s(q - z)

Here, x is the original value, q is the stored integer, s is the scale, z is the zero-point, and x_hat is the approximate value used during computation. In symmetric quantization, the zero-point is usually zero. In asymmetric quantization, it allows the representable range to be shifted.

Suppose a group of weights lies between -1.0 and 1.0, and an INT8 quantizer uses a symmetric scale of 1.0 / 127. The original value 0.2 becomes approximately code 25, and reconstructs as 25 / 127, or about 0.19685. The error is small. With only 16 INT4 codes across a comparable range, the gaps between representable values are much larger, so rounding error is more visible.

The scale can apply to an entire tensor, one channel, or a small group of weights. A single tensor-wide scale is cheap to store but can be dominated by one unusually large outlier. Per-channel or per-group scales fit the local ranges better. A common group size is 128 weights, although the right choice depends on the model and kernel. Smaller groups usually preserve quality better because they adapt to the data, but they require more metadata and may reduce kernel efficiency.

A useful notation is W4A16: weights use 4-bit integers while activations remain 16-bit floating point. W8A8 means both weights and activations use 8-bit representations. Weight-only quantization is usually easier to make accurate because activations still have plenty of numerical range. Quantizing activations as well can improve arithmetic efficiency, but it needs representative calibration data and stronger hardware support.

Common mistake: FP16 and BF16 are reduced-precision floating-point formats, not integer formats. In casual engineering conversation they are often grouped under “quantization” because they reduce precision and memory compared with FP32. Strictly speaking, INT8 and INT4 use integer codes, while FP16 and BF16 use floating-point codes with fewer bits.

Why quality can change

Quantization affects quality for three main reasons.

First, rounding changes values. The error may be tiny for an individual weight but accumulate through many matrix multiplications and residual connections. A model does not care only about the average weight error. It cares whether the error changes an attention score, a routing decision, or the probability of a rare but important token.

Second, clipping loses outliers. Suppose most values fall between -0.2 and 0.2, but one value is 4.0. If one scale covers the entire tensor, the quantizer must reserve range for 4.0, leaving fewer useful codes for the small values. Per-channel or per-group scaling can isolate that outlier instead.

Third, some parts of a model are more sensitive than others. Layer normalization, embeddings, attention projections, and the final language-model head may react differently to low precision. That is why production schemes often keep selected layers, norms, or the output head in FP16 or BF16 while quantizing the bulk of the weights. This is not a universal list; sensitivity must be measured for the particular checkpoint.

Post-training quantization, or PTQ, converts an already-trained model. Calibration means running representative examples through the model to estimate ranges and identify sensitive values. GPTQ uses calibration examples and an approximation of curvature to choose weight rounding that minimizes layer-output error. AWQ uses activation statistics to identify salient weights and protect their effect during weight-only quantization.

Quantization-aware training, or QAT, simulates quantization during training so the model can adapt to the errors. QAT can recover quality when PTQ is not good enough, but it costs additional training time and data. It is usually justified when the model is business-critical or when a very low-bit deployment target is mandatory.

A concrete example

The memory calculation is reproducible:

params = 8_000_000_000

for name, bits in [("FP32", 32), ("FP16", 16), ("INT8", 8), ("INT4", 4)]:
    gb = params * bits / 8 / 1_000_000_000
    print(f"{name}: {gb:.1f} GB")

The output is:

FP32: 32.0 GB
FP16: 16.0 GB
INT8: 8.0 GB
INT4: 4.0 GB

Now imagine evaluating a support chatbot on 1,000 held-out questions. The FP16 baseline answers 920 correctly. A W4A16 version answers 914 correctly, reduces p99 latency from 220 milliseconds to 170 milliseconds, and fits twice as many concurrent requests. That may be an excellent trade.

But if the six missed answers include account-closure policy or medication instructions, the raw score hides the real risk. I would break quality down by intent, language, context length, refusal behavior, and safety-critical cases. Quantization is acceptable only if the errors fit the product’s tolerance.

How I would use it in production

Start with a baseline. Measure the unquantized model on the exact checkpoint, tokenizer, prompts, context lengths, and serving engine. Record quality, peak memory, time to first token, tokens per second, and p99 latency, meaning the latency below which 99 percent of requests finish.

Choose the least aggressive format that solves the constraint. If FP16 cannot fit, try weight-only INT8 or W4A16 before immediately reaching for more complicated activation quantization. If the target is a supported CPU or accelerator, INT8 may deliver better real latency than INT4. The hardware kernel matters as much as the bit count.

Calibrate with production-shaped data. Include short and long prompts, code if users submit code, important languages, tool calls, and the longest contexts the service accepts. A calibration set made only of short English questions can produce a model that looks healthy offline and behaves strangely in production.

Evaluate before benchmarking claims. Compare exact task metrics and curated generations. Check rare entities, numbers, structured output, refusal behavior, and long-context retrieval. Then benchmark at realistic concurrency. A quantized model that saves memory but has worse p99 latency on the actual serving stack has not solved the operational problem.

The key-value cache deserves separate attention. It stores past attention keys and values and grows with sequence length and batch size. Quantizing the weights may make the model fit while the cache still exhausts GPU memory. Quantizing the cache can extend context or concurrency, but it introduces another quality trade-off, particularly on long conversations.

The senior-level trade-off and a failure mode

Lower precision is not automatically faster. If the runtime has an optimized INT4 kernel, reduced memory traffic may help substantially. If it has to unpack integers and dequantize them inefficiently, the extra work can erase the gain. Quantization can still be worthwhile purely because the smaller model fits, allowing a larger batch or avoiding an expensive GPU, but that is a capacity benefit rather than a guaranteed per-request speedup.

A common failure appears first as a quality incident, not a deployment error: offline tests pass, but production answers become repetitive, rare names are corrupted, or accuracy falls sharply on long prompts. The usual suspects are a calibration set that does not match production, activation outliers, an overlarge quantization group, or a sensitive layer that was quantized too aggressively. I would compare the failing examples with calibration coverage, inspect per-layer error where tooling allows it, and test selective higher precision before abandoning quantization entirely.

What they’ll ask next

Does INT4 always reduce latency?
No. It reduces memory traffic and model size, but latency depends on batch size, context length, device, and whether the serving engine has an efficient INT4 kernel. Benchmark the target workload rather than inferring speed from the file size.

How would you choose between INT8 and INT4?
I would begin with the quality and memory constraints. INT8 usually provides more numerical headroom; INT4 saves more memory but needs better scaling, calibration, and sometimes selective higher-precision layers. I would accept the most aggressive format that passes task, safety, and latency gates.

Is quantization the same as pruning or distillation?
No. Quantization reduces the representation precision of existing values. Pruning removes some weights or connections, while distillation trains a smaller model to imitate a larger one. They can be combined, but they create different failure modes and require separate evaluation.

Say this in the interview

“Quantization trades representation precision for memory and bandwidth; I would choose a calibrated format such as W8A16 or W4A16 only after measuring task quality, safety, and latency on the target hardware, while treating sensitive layers and the KV cache separately.”