MHA, MQA, GQA, and MLA: attention efficiency trade-offs

At 3 a.m., the fifth customer request arrives with a 96,000-token context. The first four long requests were fine. This one makes the GPU throw CUDA out of memory. Nothing about the model weights changed. The prompt is not unusual. The model simply has to remember five long conversations at once.

That is the part of LLM serving that parameter counts hide.

My view is simple: MHA, MQA, GQA, and MLA are not minor attention variants to memorise for an architecture quiz. They are different storage formats for the model’s memory of the past.

The choice often decides whether a GPU serves two long-context users or twenty. It can determine whether decoding is smooth or stalls, and whether a marketed context window is usable in production.

The useful story is how different designs store that memory, and which trade each makes among cache size, quality, compute, and implementation complexity.

The cache grows one token at a time

A transformer processes a prompt in a phase called prefill, where it reads the existing tokens and builds internal representations. It then generates the answer one token at a time in decode.

For each layer and each token, attention creates a key, usually written K, and a value, written V. The key says what a token can be matched against. The value contains what should be retrieved if that token is attended to.

During decoding, the new token’s query compares itself with the keys from every earlier token. It then uses the resulting weights to combine their values.

Recomputing all those old keys and values for every new token would be wasteful. So the serving system stores them in the KV cache. The cache is the model’s working memory for the current request.

It grows with context length, number of layers, and number of cached attention heads. It also grows with the number of active requests.

At long context and high concurrency, the cache can dominate dynamic GPU memory and become a bandwidth bottleneck during decoding. The GPU is excellent at large matrix multiplications. It is less thrilled about repeatedly streaming gigabytes of history through memory to produce one more token.

A first-order cache estimate

For one sequence, a useful estimate is:

KV bytes = 2 × L × T × H_kv × d_h × s

Here:

• L is the number of transformer layers
• T is the number of cached tokens
• H_kv is the number of key/value heads
• d_h is the head dimension
• s is the number of bytes per stored number

The first 2 accounts for both keys and values.

Take a model with 32 layers, 32 query heads, head dimension 128, an 8,192-token context, and an fp16 or bf16 cache. The cache sizes are:

layers = 32
tokens = 8192
head_dim = 128
bytes_per_number = 2  # fp16 or bf16

def cache_gib(kv_heads):
    bytes_ = (
        2
        * layers
        * tokens
        * kv_heads
        * head_dim
        * bytes_per_number
    )
    return bytes_ / (2 ** 30)

for name, kv_heads in [("MHA", 32), ("MQA", 1), ("GQA", 8)]:
    print(name, f"{cache_gib(kv_heads):.3f} GiB")

The output is 4.000 GiB for MHA, 0.125 GiB for MQA, and 1.000 GiB for GQA. That is one sequence, before page rounding, allocator overhead, temporary buffers, replicated copies, or other model state.

At 128,000 tokens, multiply each number by 16. The same sequence needs roughly 64 GiB with MHA, 16 GiB with GQA, or 2 GiB with MQA.

That is why the attention variant is an infrastructure decision, not a footnote in a model card.

MHA: every query head gets its own memory

Multi-head attention, or MHA, is the traditional design. The model creates several query heads, several key heads, and several value heads. Each query head has a matching key/value head.

With 32 heads, the cache stores 32 sets of keys and 32 sets of values for every token at every layer. In the formula above, H_kv equals 32.

Why keep them separate? Because each head can learn a different way to compare and retrieve information. For example, one head may be useful for:

• nearby syntax
• tracking a subject across a paragraph
• focusing on delimiters or code structure

Those roles are not assigned by hand, and they are not perfectly distinct. Separate projections give the model room to develop different attention subspaces.

MHA is therefore the quality and flexibility baseline. It is also the expensive baseline. In our example, each token contributes:

2 × 32 × 128 = 8,192

cached numbers per layer. The cache is not large because the query is large. The query for the next token is temporary.

The cache is large because every old key and value is retained.

MQA: keep the queries, share the memory

Multi-query attention, or MQA, keeps all 32 query heads but gives them one shared key head and one shared value head. In the formula, H_kv falls from 32 to 1.

That cuts the cache by 32 times. It is a spectacular memory saving.

MQA is not the same as reducing the model to one attention head. The 32 query heads still produce different queries. They can therefore assign different attention weights to the same sequence of cached positions.

Query head 3 and query head 19 may look at different tokens.

The restriction is subtler. Both heads must draw their information from the same key representation and the same value representation.

They no longer have separate value subspaces in which to store and retrieve different kinds of information. If two heads want different notions of relevance or different ways of combining retrieved content, they have to share the same underlying K/V basis.

That is where a quality trade-off can appear. MQA does not make every head look at the same place. It makes every head search the same kind of memory.

The quality loss is not a law of nature. A model trained from the beginning with MQA can learn to use the shared representation well, and some workloads may barely care.

But taking an MHA checkpoint and crudely collapsing its K/V heads is not free. Long-context retrieval, code completion, and exact instruction following are the sorts of behaviours that can quietly degrade while ordinary short answers remain fluent.

With one shared K/V pair, MQA is the extreme case of the grouping idea: every query head belongs to the same group.

MQA proved that the cache could be slashed. It also demonstrated that the most aggressive saving can ask the model to give up too much representational freedom.

GQA: share in groups, not universally

Grouped-query attention, or GQA, puts several query heads in a group and gives each group one key/value head.

Return to the 32-query-head example. With 8 K/V heads, each K/V head serves 4 query heads:

• query heads 0 through 3 share K/V head 0
• query heads 4 through 7 share K/V head 1
• the pattern continues through all 32 query heads

The cache is now four times smaller than MHA because H_kv is 8 rather than 32. The query side remains 32-headed.

The model retains much more K/V diversity than MQA while avoiding the full MHA bill.

That is why GQA is often the pragmatic quality/cache compromise in modern open-weight models. It captures much of MHA’s quality at a substantially lower memory cost.

The exact ratio is not sacred. A model might use 4, 8, or another number of K/V heads, provided the architecture and implementation support it.

GQA does not make decoding four times faster automatically. The cache is four times smaller, and K/V memory traffic can fall substantially, but the system still computes attention for all 32 query heads.

A serving kernel may broadcast a K/V head across its group without materialising four separate copies, or it may use a less efficient fallback. The cache ratio is a strong architectural clue, not a guaranteed wall-clock benchmark.

The other important point is that these head counts are normally part of the trained checkpoint. They are not a harmless runtime switch.

A serving engine cannot generally turn an MHA model into a high-quality MLA model by changing one setting. MHA-to-GQA conversion is possible through techniques such as pooling K/V projections and further training, but the result must be evaluated as a changed model.

The baseline mechanics are covered in multi-head attention. GQA is best understood as changing only the K/V side of that design.

MLA: compress the memory itself

Multi-head latent attention, or MLA, takes a different route. It is a separate low-rank approach, meaning it stores a lower-dimensional learned representation instead of every full K/V vector.

MQA and GQA ask how many K/V heads to keep. MLA asks whether the full K/V vectors need to be stored at all.

For a token with hidden state h_t, a learned down-projection creates a smaller latent vector:

c_t = W_down h_t

The cache stores c_t, rather than storing every full key and value. Learned up-projections can use that latent to recover the information needed for attention.

In a DeepSeek-style design, there is also a separate positional key component for rotary positional information. The exact dimensions depend on the model.

The important trick is that these are learned linear projections. The implementation can rearrange and absorb parts of the up-projection into the query and output calculations.

It does not necessarily reconstruct a complete, materialised K/V tensor for every head at every step. Logically, the information needed by the different heads remains available through the latent representation. Physically, the cache stores a much smaller object.

A concrete DeepSeek-V2/V3-style configuration uses a 512-number compressed KV latent plus a 64-number positional key. That is 576 cached latent and positional numbers per layer and token.

These dimensions are an illustration, not a universal MLA specification. The latent width, positional component, head dimension, and layer count vary by architecture.

Compare that with the earlier example:

Design	Cached numbers per layer, token	Relative cache
MHA, 32 K/V heads	8,192	1 times
MQA, 1 K/V head	256	32 times smaller
GQA, 8 K/V heads	2,048	4 times smaller
MLA, illustrative latent	576	about 14 times smaller

Those MLA numbers are illustrative, not a universal MLA specification. The positional component, latent width, head dimension, and layer count vary by architecture.

Still, the mechanism is clear: in this example, the latent cache is about 3.6 times smaller than the 8-head GQA cache, but larger than MQA. It stores 576 rather than 256 cached numbers per layer and token.

The saving is not magic. MLA moves work from storage and memory bandwidth into projection and attention computation. It also demands kernels that understand the layout.

On a long-context, highly concurrent decode workload, that trade can be excellent: memory capacity and bandwidth are scarce, while a little extra arithmetic is affordable.

On short prompts or a deployment with weak MLA support, the extra transformations can erase the benefit.

MLA also does not mean “no cache”. The latent still has one entry per token, and the positional information still grows with context. It changes the slope of memory growth. It does not turn linear growth into constant memory.

MLA is designed to preserve useful attention quality while reducing cache size, but there is no architecture theorem saying that every MLA model will achieve MHA-like quality.

The latent dimensions are learned as part of training. A strong native MLA checkpoint and an improvised compression of an unrelated MHA checkpoint are very different things.

These designs are not a quality leaderboard

The four designs can be summarised as four answers to one engineering question:

Design	What is shared or compressed	What it buys
MHA	Nothing across heads	Maximum K/V flexibility
MQA	One K/V pair for all queries	Minimum ordinary K/V cache
GQA	One K/V pair per query group	Strong memory-quality compromise
MLA	A learned latent plus positional state	Configurable compressed cache; more projection work

MQA is the extreme case of GQA: one K/V head serves every query head. GQA is the pragmatic quality/cache compromise: it keeps multiple K/V groups rather than one or one per query.

MLA is a separate low-rank approach, not a later GQA setting. Its cache size depends on its latent and positional dimensions.

In our worked example, moving from MQA to GQA changes H_kv from 1 to 8, so the cache becomes eight times larger. The point of GQA is the quality/cache compromise, not a monotonic reduction in bytes.

People often present MHA, MQA, GQA, and MLA in this historical/design-space order. That order is not a ranking by cache size or quality.

A well-trained MQA model can be better for a particular job than a poorly trained GQA model. A native MLA model may be cheaper to serve than GQA at long context, but more awkward to integrate.

There is also a strong objection to the whole argument: the KV cache is not always the bottleneck. For a short prompt at batch size one, prefill matrix multiplications, weight loading, or the model’s ordinary compute may dominate.

Reducing K/V heads may barely change time to first token. If MHA fits comfortably and its quality matters, MHA can be the sensible choice.

Other ways to manage cache pressure

Other techniques attack the same problem from different directions:

• Changing the cache from bf16 to an 8-bit format can roughly halve its raw storage, though scales, metadata, kernels, and accuracy complicate the result.
• Offloading moves cache pressure from GPU memory to CPU memory or another device, but the transferred data still has to travel somewhere.
• Paging reduces fragmentation and lets a server share memory more effectively; it does not reduce the number of bytes needed for unique history.
• Prefix caching avoids storing duplicate prefixes when requests genuinely share them, but it does not shrink each request’s unique suffix.

That is why KV-cache offloading is a useful fallback, not a substitute for choosing a sensible attention architecture. The right answer depends on whether the production constraint is capacity, bandwidth, compute, quality, or engineering support.

What to do on Monday morning

Start with the checkpoint, not the model’s advertised context window. Record:

• the number of layers
• query heads
• K/V heads
• head dimension
• cache datatype
• for MLA, the latent and positional dimensions

Then calculate the cache for the actual number of simultaneous sequences. The single-sequence estimate is multiplied by both concurrency and the tokens retained per request.

Next, test the service shape you actually expect. Try prompt lengths such as 2,000, 8,000, and 32,000 tokens. Then test active-request counts such as 1, 8, and 32 where the hardware allows it.

Measure:

• time to first token
• the time between generated tokens
• peak GPU memory
• p95 and p99 latency
• rejected requests
• output quality

A model that looks cheap at one request may become unusable when eight users arrive with long histories.

Use a paged KV allocator and continuous batching when your serving stack supports them. Add an input-token limit, an output-token limit, and admission control before the GPU is already full.

Enable prefix caching only when requests share exact or safely reusable prefixes. The practical mechanics belong with broader LLM serving operations, because cache size is only one part of a live serving system.

If you are choosing a new model for a general deployment, GQA is usually the safest middle ground: meaningful cache savings without the most aggressive sharing.

Choose MQA when the target model was trained for it and your quality tests show that the reduction is acceptable. Choose MLA when the model is natively built around it and the inference engine has a good implementation.

Do not select by the acronym alone. Compare the total cost and latency for your real request distribution, as discussed in LLM cost and latency.

The failure modes show up before the formula does

The first symptom of a cache problem is often not an immediate out-of-memory error. Decode latency starts climbing as conversations get longer.

The first token still arrives quickly, but the gap between later tokens widens. At higher concurrency, the p99 becomes ugly while average latency looks respectable.

If the crash appears only on the second or third long request, check batch × tokens, not just the model’s maximum context. Include:

• prompt tokens already prefetched
• generated tokens
• beam copies or duplicated sequences if your application uses them

The formula gives a lower-bound estimate. The allocator and runtime need additional room.

A different failure appears after an MHA-to-MQA or MHA-to-GQA conversion. Answers remain grammatical, but the model misses a detail buried near the start of a long document, calls the wrong tool, or loses a code constraint.

That is a quality failure in the shared K/V representation, not a serving outage. Compare retrieval and instruction-following tests before and after conversion.

The final lesson is the one model cards tend to leave implicit. Parameter count tells you a great deal about weight memory and training scale.

The K/V design tells you how expensive the model’s memory of the conversation will be. MHA sets the quality baseline. MQA shows how far ordinary K/V caching can shrink. GQA offers the useful quality/cache compromise. MLA compresses the stored representation itself, with a cache size determined by its configuration.

The cheapest model to train is not automatically the cheapest model to keep alive at 3 a.m.