How do attention masks and pretraining objectives influence the choice between BERT, GPT, and T5?

How to think about it

The direct answer is that encoder-only models read an input in both directions, decoder-only models read from left to right, and encoder-decoder models do both: they encode the full input and then generate an output left to right while looking back at that encoded input. BERT is the classic encoder-only example, GPT-style models are decoder-only, and T5 and BART are encoder-decoder models.

The important difference is not the name. It is the information each position is allowed to see.

Why the attention pattern matters

A transformer processes text as tokens, which are small text units such as words, word pieces, or punctuation. In self-attention, each token creates a query, meaning what it is looking for; a key, meaning what it offers for matching; and a value, meaning the information it passes along. A token compares its query with other tokens’ keys, then combines their values.

A mask is a rule that blocks particular attention connections. A blocked connection contributes nothing, so the model cannot use information it should not have.

For four token positions, the visibility patterns look like this. A one means “visible”; a zero means “blocked”. Each row asks what one position can read from the columns.

Encoder
1 1 1 1
1 1 1 1
1 1 1 1
1 1 1 1

Decoder
1 0 0 0
1 1 0 0
1 1 1 0
1 1 1 1

An encoder uses full, bidirectional self-attention. “Bidirectional” means information can flow from earlier positions to later positions and from later positions back to earlier ones. It does not mean that the model runs two separate encoders.

A decoder-only model uses causal self-attention, meaning position i can read position i and positions before it, but not positions after it. This prevents the model from seeing the answer while learning to predict the answer.

An encoder-decoder model has two attention paths. Its encoder uses full attention over the source, meaning the input being transformed. Its decoder uses causal self-attention over the target, meaning the output being produced, plus cross-attention, where each decoder position reads the encoder’s representations of the source.

That division is the whole story in miniature: understand the complete input, then produce the output without peeking at future output tokens.

A concrete example

Suppose a support system receives this text:

The package arrived late, but the box was intact.

An encoder-only model can build a representation of the entire sentence. The representation for late can use package, arrived, box, and intact. The representation for intact can use late as well.

A small classification head, meaning a neural layer that converts the representation into labels, might predict:

• delivery problem
• damaged item: no
• sentiment: negative

The same encoder can produce a dense embedding, meaning a vector designed to capture meaning. A search system could compare that vector with other support tickets and find similar complaints.

A decoder-only model sees the sentence as a prefix and predicts what comes next. It might produce:

The customer should receive a delivery-fee refund.

It generates one token, appends that token to the context, and predicts the next one. This is autoregressive generation: each new token is conditioned on the tokens already available.

An encoder-decoder model first encodes the complete support ticket. Its decoder can then generate a target such as a short summary:

Late delivery; package undamaged.

At every generated position, the decoder can read the source representation through cross-attention. It cannot read future summary tokens, because those do not exist yet.

The numbers make the difference less abstract. Suppose the source has six token positions and the generated target has four:

• Full encoder self-attention considers 6 × 6 = 36 query-key connections.
• A causal decoder over six positions permits 6 × 7 / 2 = 21 connections.
• An encoder-decoder decoder over four target positions permits 4 × 5 / 2 = 10 target self-attention connections and 4 × 6 = 24 source cross-attention connections.

Those counts describe one attention head in one layer and ignore padding. They are not benchmark timings. They show exactly what the mask permits.

Why the pretraining objectives usually differ

The architecture and the pretraining objective normally fit each other.

An encoder-only model is commonly trained with masked language modelling, or MLM. Some input tokens are replaced with a mask, and the model predicts them from the surrounding context:

The package arrived [MASK], but the box was intact.

The model can use words on both sides of the missing token. That encourages each token representation to incorporate its whole sentence. BERT and RoBERTa are well-known examples.

A decoder-only model is commonly trained with next-token prediction. Given:

The package arrived

it learns to assign a high probability to the next token, perhaps late. The causal mask prevents it from reading later tokens during that prediction.

There is a subtle but important point here: causal training does not require the hardware to process one token at a time. During training, all positions in a sequence can be processed in parallel. The mask hides future positions, and the labels are shifted so each position predicts the next one. At inference time, however, the answer is not known in advance, so generation is sequential.

An encoder-decoder model is usually trained on a sequence-to-sequence objective, meaning it maps one text sequence to another. T5 corrupts spans of text and trains the decoder to reconstruct the missing spans. BART corrupts an input and trains a causal decoder to reconstruct the original sequence. Translation, summarisation, and structured rewriting fit this setup naturally.

These objectives are conventions, not prison walls. An encoder can be fine-tuned for contrastive embeddings. A decoder-only model can be instruction-tuned to classify text. An encoder-decoder model can answer questions, rewrite documents, or generate code.

How I would choose one

For sentence classification, named entity recognition, extractive question answering, and semantic search, I would start with an encoder-only model or a model specifically trained for embeddings. It produces a representation in one forward pass rather than generating a label token by token. For retrieval, a bi-encoder, meaning a system that encodes the query and document separately, lets the system precompute document vectors.

For chat, code completion, agents, and open-ended writing, I would start with a decoder-only model. Its training objective directly matches continuation, and the surrounding tooling for serving, instruction tuning, tool calling, and in-context learning is extensive.

For translation, summarisation, and controlled source-to-target transformations, an encoder-decoder model is the canonical fit. The source is read bidirectionally once, while the target is generated with the right causal constraint.

There is a practical trade-off. An encoder-only model is usually the simplest and fastest choice when the output is just a label or vector. A decoder-only model has a single repeated stack and a very broad ecosystem, but generating a 300-token answer requires a generation loop. An encoder-decoder model has two stacks and cross-attention, but it cleanly separates source understanding from target generation and can be a strong fit when the input and output have different jobs.

The right answer depends on output length, latency targets, available checkpoints, fine-tuning data, memory limits, and evaluation results. Architecture is a prior, not a guarantee.

The nuance that earns the senior signal

Do not say that encoder-only models “cannot generate” without qualification. A standard BERT-style model has no autoregressive decoder, so it does not naturally generate a sentence from left to right. It can fill masks iteratively, but that is a different generation procedure and is not equivalent to normal causal decoding.

Do not say that decoder-only models cannot understand a complete prompt. During generation, the final prompt position can attend to every earlier prompt token, so the next-token decision uses the whole prompt. But each prompt position still obeys the causal mask. Earlier positions cannot revise themselves after seeing later positions, and generated tokens cannot see future generated tokens.

Finally, the architecture alone does not determine embedding quality. A raw hidden state from a decoder-only language model may be a poor sentence embedding because next-token training does not directly teach paraphrases to land near one another. A decoder-only model that has been explicitly fine-tuned as an embedding model can perform very well. The training objective, pooling method, data, and model scale matter.

What they’ll ask next

“Why can’t BERT generate text like GPT?”

BERT was trained to reconstruct masked tokens using both sides of the input. It was not trained to maintain a left-to-right generation state. GPT has a causal mask and a next-token output distribution, so it can append one predicted token and continue. BERT can be adapted for generation, but it needs an additional decoding strategy or a different architecture.

“Does a decoder-only model see the whole prompt?”

The last prompt position can see the entire prompt to its left, so it can use the full prompt when predicting the first new token. Other prompt positions cannot see tokens to their right. The model understands the prompt through the representation at the positions used for prediction, not because every position has bidirectional visibility.

“Which architecture would you use for semantic search and translation?”

For semantic search, I would begin with an encoder or a dedicated embedding model because queries and documents need reusable vectors. For translation, I would begin with an encoder-decoder model because the source and target have distinct roles. I would still benchmark a decoder-only model if its quality, multilingual coverage, serving stack, or instruction-following behaviour justified the extra generation cost.

For the longer BERT, GPT, and T5 comparison, see BERT, GPT, and T5.

Say this in the interview: “Encoder-only models build bidirectional representations, decoder-only models generate left to right with causal masking, and encoder-decoder models encode the full source before generating a target through cross-attention; I choose among them based on whether the task needs representation, continuation, or source-conditioned transformation.”