What is mixed precision training and why does it matter?

How to think about it

Mixed precision training uses float16 or bfloat16 for the expensive tensor math and float32 for calculations and state that need more numerical range or precision. It matters because lower-precision tensors use half the memory and can run much faster on hardware such as NVIDIA Tensor Cores, but using low precision everywhere can make training overflow, underflow, or silently stop learning.

Why it matters

Training a neural network creates many intermediate tensors. An activation is one of those intermediate results, such as the output of a layer, that must often be saved so backpropagation can calculate gradients later. A large transformer can spend more GPU memory on these saved activations than on its parameters.

A float32 value occupies 4 bytes. A float16 or bfloat16 value occupies 2 bytes. If an activation tensor contains 1 billion values, that is roughly 4 GB in float32 and 2 GB in a 16-bit format. The saving is real, although the whole training job will not necessarily use half as much memory because parameters, gradients, optimizer states, and some temporary results may remain in float32.

There is also a speed benefit. GPUs contain specialized matrix-multiplication hardware called Tensor Cores. Matrix multiplications and convolutions in float16 or bfloat16 can use these units efficiently. A well-shaped workload may therefore gain substantially more throughput than the same workload using ordinary float32 arithmetic.

That last sentence contains an important qualification: mixed precision is not automatically a two-times speedup. If the input pipeline, communication, memory movement, small matrix sizes, or CPU preprocessing is the bottleneck, changing the arithmetic format may barely move the final tokens-per-second number.

The formats make the trade-off clearer:

Format	Bytes per value	Main characteristic
float32	4	Broad range and about 7 decimal digits of precision
float16	2	More precision than bfloat16, but a narrow numerical range
bfloat16	2	A float32-like exponent range, but fewer significant bits

float16 has a largest finite value of 65,504, and its smallest normal positive value is about 6.10e-5. Smaller subnormal values exist, but they have very few useful bits, and some hardware kernels handle them less generously. A small gradient or a chain of multiplications can therefore become zero.

bfloat16 keeps the wide exponent range of float32. It is much less likely to overflow or underflow merely because a value is very large or very small. Its weakness is precision: around a value such as 1.0, it cannot distinguish changes as finely as float16 or float32.

What actually happens during training

The common beginner description is “store the model in float16, then update it in float32.” That describes one valid implementation, but not all of them.

There are two patterns worth separating:

1. In a pure half-precision implementation, the model may have float16 weights used for computation and a separate float32 master copy used by the optimizer.
2. In PyTorch automatic mixed precision, or AMP, model parameters commonly remain in float32. Autocasting chooses lower precision for eligible operations, such as many matrix multiplications, while keeping other operations in float32.

The second pattern is why “mixed precision” does not mean “every tensor is half precision.” Autocast is an operation-level policy, not a global cast. Reductions, normalization, softmax-like operations, loss calculations, and accumulations may use float32 because rounding errors can compound there. Backpropagation follows the choices made by the forward operations; it is not guaranteed to run entirely in float16.

Keeping the optimizer update in float32 matters even when gradients are small but representable. Suppose a weight is 1.0 and the update is 1e-7. Near 1.0, adjacent float16 values are about 0.0009765625 apart. Adding 1e-7 to the weight in float16 rounds back to 1.0; the update disappears. A float32 master weight can represent the change and accumulate many such updates over time.

Loss scaling for float16

float16 training commonly uses loss scaling, which multiplies the loss by a temporary scale factor before backpropagation. If the true gradient contains a component of 1e-8, multiplying the loss by 65,536 makes the corresponding backward value about 6.5536e-4. That value is much easier for half-precision operations to preserve. The framework later divides the gradients by the same scale before the optimizer update.

The scale cannot be chosen blindly. If it is too large, a gradient can exceed float16’s maximum and become infinity. PyTorch’s GradScaler watches for non-finite gradients, skips an unsafe optimizer step, and adjusts the scale. In a healthy run, the scale usually grows when training is stable and backs off when overflow occurs.

A current PyTorch-style float16 loop looks like this:

import torch

scaler = torch.amp.GradScaler("cuda")

for inputs, labels in loader:
    optimizer.zero_grad(set_to_none=True)

    with torch.autocast(device_type="cuda", dtype=torch.float16):
        logits = model(inputs)
        loss = loss_fn(logits, labels)

    scaler.scale(loss).backward()

    # Required before clipping or inspecting the true gradient values.
    scaler.unscale_(optimizer)
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    scaler.step(optimizer)
    scaler.update()

The optimizer step is deliberately outside the autocast block. Gradient clipping is also after unscale_; clipping the scaled gradients would use the wrong norm.

With bfloat16, loss scaling is usually unnecessary because its exponent range is so wide. A typical loop uses autocast with dtype=torch.bfloat16, then calls ordinary loss.backward() and optimizer.step(). “Usually” is doing honest work here: bfloat16 prevents many range failures, but it does not make every model numerically stable.

A concrete memory example

Suppose you are training a 1-billion-parameter transformer with Adam on a 40 GB GPU.

Ignoring activations, a rough float32 memory budget is:

• Parameters: 1 billion times 4 bytes, or about 4 GB
• Gradients: about 4 GB
• Adam’s first and second moments: about 8 GB total

That is already about 16 GB before temporary buffers, CUDA workspace, communication buffers, or activations.

Assume profiling shows that the chosen batch size and sequence length require another 18 GB for saved activations. The rough total is then 34 GB, leaving only 6 GB of headroom.

With AMP, many eligible activations are stored in a 16-bit format. If that activation portion falls from roughly 18 GB to 9 GB, the comparable total is about 25 GB. The GPU now has much more room for a larger batch, longer sequence, or temporary workspace.

It is not accurate to conclude that a model twice as large will fit. In the common AMP setup, the 16 GB of persistent parameter, gradient, and Adam state memory has barely changed. The gain came mainly from activations. If the model is optimizer-state-bound rather than activation-bound, mixed precision will disappoint as a memory solution.

A larger batch can also change optimization behaviour. It may alter gradient noise, learning-rate scaling, and generalization. More capacity is useful only if the rest of the training recipe still makes sense.

The senior-level nuance

float16 and bfloat16 are not interchangeable.

Choose float16 when the hardware supports it well and the model benefits from its somewhat finer precision. Be prepared to use loss scaling and to investigate overflow or underflow.

Choose bfloat16 when the hardware supports it efficiently and the model has large activation or gradient ranges. Large language models often prefer it because the float32-like exponent range removes much of the loss-scaling machinery. The cost is coarser precision, which can matter in sensitive reductions or models already close to numerical instability.

For either format, keep numerically sensitive pieces in float32 when necessary. A model may use lower precision for its large linear layers while calculating a normalization statistic or final loss in float32. That is still mixed precision, and it is often the production pattern.

The same idea now extends to formats such as FP8 in some training systems. FP8 is a more aggressive trade-off, usually requiring explicit scaling metadata, suitable hardware, and a validated recipe. It is not a drop-in replacement for ordinary FP16 or BF16 AMP.

Do not use mixed precision simply because a benchmark says it is fashionable. On unsupported hardware, conversion overhead can outweigh the benefit. For a small model, training may already be dominated by data loading. For a numerically fragile model, start with a float32 baseline so you can distinguish an algorithmic problem from a precision problem.

A failure mode you will actually see

The obvious failure is a loss that suddenly becomes nan or inf, often after a learning-rate warmup step or a batch containing unusually large values. In a scaled float16 run, you may also see the scaler value repeatedly decrease and optimizer steps get skipped.

First check that the loss and gradients are finite. Log the scale, inspect gradient norms, and identify the first non-finite activation if possible. Then try bfloat16 if the hardware supports it, keep the sensitive operation in float32, reduce the loss scale, or fix the underlying data or learning-rate problem.

The quieter failure is a finite loss that stops improving. Gradients may contain many exact zeros because small values vanished during half-precision computation. Loss scaling, BF16, or a selective FP32 path can fix that. Turning off all numerical checks merely produces a more convincing-looking broken model.

What they’ll ask next

Does mixed precision mean the weights are stored in float16?

Not necessarily. With PyTorch AMP, parameters commonly stay in float32, while eligible operations receive lower-precision inputs. A separate FP32 master copy is more characteristic of pure FP16 implementations. The invariant is that the optimizer must have a sufficiently precise state for accumulating updates.

Why choose BF16 over FP16?

BF16 has the same exponent width as FP32, so it handles very large and very small values much better. FP16 has more mantissa bits, so it offers finer precision near ordinary values but has a much narrower range and usually needs loss scaling. The choice depends on hardware support and the model’s numerical behaviour.

What would you monitor in production?

I would compare validation metrics against a float32 or trusted BF16 baseline, then track step time, samples or tokens per second, peak GPU memory, non-finite losses, skipped optimizer steps, gradient norms, and the loss-scale history for FP16. A faster run that silently loses accuracy is not an optimization.

Say this in the interview

Mixed precision uses FP16 or BF16 for tensor-heavy work and FP32 for sensitive calculations and optimizer state, giving lower memory use and often higher throughput; FP16 needs loss scaling, and the actual speed and accuracy gain must be measured on the target model and hardware.