BERT, GPT, T5 — picking the right architecture

Three transformer families exist, each built around a different pretraining objective. The objective shapes everything — what tasks the model is good at, how you use it, and how big it tends to be. Knowing which family fits your problem saves you both money and embarrassment.

The cheat sheet

BERT — the encoder for understanding

BERT (2018) processes the whole input bidirectionally — every token sees every other token, in both directions. Pretraining task: “I’ll mask 15% of the tokens; predict them.” To do this well, the model has to deeply understand context in both directions.

"The [MASK] sat on the mat."
              ↓
"The cat sat on the mat."

BERT is not generative — bidirectional attention means it can see the whole sequence at once, so it can’t autoregressively produce output token-by-token. BERT-style models excel at understanding tasks where you need a fixed-size representation of a piece of text:

• Sentence classification (sentiment, spam, intent)
• Token classification (named entity recognition)
• Sentence-pair tasks (similarity, entailment)
• Embeddings for retrieval — this is where they shine today

Embeddings is the killer app for encoder-only models. Every RAG system uses an encoder to turn documents and queries into vectors. Models like BAAI/bge, intfloat/e5, nomic-ai/nomic-embed, and mixedbread-ai/mxbai-embed-large are BERT-family descendants optimized for retrieval — and they’re the workhorse of modern LLM applications.

GPT — the decoder for generation

GPT-style models use causal attention: token i can only see tokens 0..i. The pretraining task is “predict the next token, given all the previous ones.”

"The cat sat on the" -> "mat"

Causal attention is what makes generation natural — each new token is predicted from the previous ones, then appended, then the process repeats. Decoder-only models are the basis of:

• Chat assistants (ChatGPT, Claude, Gemini)
• Code generation (Copilot, Cursor)
• Agentic systems
• Anything that produces variable-length output

Examples today: GPT-4o, Claude Sonnet/Opus, Llama 3.1/3.2, Mistral Large, Qwen 2.5, DeepSeek V3. Every major frontier LLM today is decoder-only. This is the dominant paradigm, and it’s not close.

T5 — encoder-decoder for seq2seq

T5 (2019) has both an encoder (bidirectional) and a decoder (causal). Pretraining: corrupt spans in the input, train the decoder to produce them.

Input:  "The <X> sat on <Y>."
Output: "<X> cat <Y> the mat"

T5 shines on seq2seq tasks: translation, summarization, structured text-to-X. T5 family today: original T5, FLAN-T5, CodeT5+, mT5. They’re not extinct, but mostly displaced by big decoder-only models that handle the same tasks via prompting.

When to pick which

Use an encoder-only model when:

• You need embeddings (retrieval, clustering, classification at scale)
• You’re doing classification on short text and want a cheap, fast model
• Your task has a fixed output shape, not free-form text

Use a decoder-only model when:

• You want generation, chat, instruction-following
• You want one model that does many tasks
• You want to use it via an API or open-weights (almost everything)

Use an encoder-decoder model when:

• You’re doing translation or summarization at scale and need every drop of performance
• You’re building on existing T5 checkpoints
• Otherwise: just use a decoder-only model

Hugging Face naming, decoded

bert-base-uncased            # encoder-only, English, lowercase
google/flan-t5-base          # encoder-decoder, instruction-tuned
meta-llama/Llama-3.1-8B      # decoder-only, base
meta-llama/Llama-3.1-8B-Instruct  # decoder-only, chat-tuned

The convention: org/model-version-size-variant. “Instruct” or “Chat” in the name = RLHF/SFT-tuned for following instructions; without it, you get the raw pretrained model.

In one breath

• Three families, three pretraining objectives, three attention masks — and the mask is the deepest difference.
• BERT (encoder-only, bidirectional, masked-LM): every token sees every other, so it understands but can’t generate — the workhorse for embeddings, retrieval, and classification.
• GPT (decoder-only, causal, next-token): token i sees only 0..i, which makes generation natural — chat, code, agents. Every frontier LLM today is decoder-only, because one simple objective scales and prompting recovers the rest.
• T5 (encoder-decoder, span-corruption): a bidirectional encoder plus a causal decoder with cross-attention — strongest on explicit seq-to-seq (translation, summarization), but mostly displaced by big decoder-only models. Rule of thumb: embeddings → encoder; generation → decoder; otherwise just use a decoder-only model — and don’t use an LLM where an embedding model belongs.

Quick check