BERT, GPT, T5 — picking the right architecture

Three transformer families exist, each built around a different pretraining objective (the practice task a model learns from before you ever use it). The objective shapes everything: what tasks the model is good at, how you use it, and how big it tends to be. Knowing which family fits your problem saves you both money and embarrassment.

Two words you will see throughout. An encoder reads the whole input and turns it into numbers that capture its meaning. A decoder writes output one token at a time. The three families use an encoder alone, a decoder alone, or both.

The cheat sheet

BERT — the encoder for understanding

BERT (2018) processes the whole input bidirectionally — every token sees every other token, in both directions. Pretraining task: “I’ll mask 15% of the tokens; predict them.” To do this well, the model has to deeply understand context in both directions.

"The [MASK] sat on the mat."
              ↓
"The cat sat on the mat."

BERT is not generative. Bidirectional attention means it sees the whole sequence at once, so it can’t autoregressively produce output (write one token at a time, each based on the ones before). BERT-style models excel at understanding tasks where you need a fixed-size representation of a piece of text:

• Sentence classification (sentiment, spam, intent)
• Token classification (named entity recognition)
• Sentence-pair tasks (similarity, entailment)
• Embeddings for retrieval — this is where they shine today

Embeddings is the killer app for encoder-only models. Every RAG system (retrieval-augmented generation, where a model looks up documents before answering) uses an encoder to turn documents and queries into embeddings: lists of numbers where similar meanings land close together. Models like BAAI/bge, intfloat/e5, nomic-ai/nomic-embed, and mixedbread-ai/mxbai-embed-large are BERT-family descendants optimized for retrieval — and they’re the workhorse of modern LLM applications.

GPT — the decoder for generation

GPT-style models use causal attention: token i can only see tokens 0..i. The pretraining task is “predict the next token, given all the previous ones.”

"The cat sat on the" -> "mat"

Causal attention is what makes generation natural — each new token is predicted from the previous ones, then appended, then the process repeats. Decoder-only models are the basis of:

• Chat assistants (ChatGPT, Claude, Gemini)
• Code generation (Copilot, Cursor)
• Agentic systems
• Anything that produces variable-length output

Examples today: GPT-4o, Claude Sonnet/Opus, Llama 3.1/3.2, Mistral Large, Qwen 2.5, DeepSeek V3. Every major frontier LLM today is decoder-only. This is the dominant paradigm, and it’s not close.

T5 — encoder-decoder for seq2seq

T5 (2019) has both an encoder (bidirectional) and a decoder (causal). Pretraining: corrupt spans in the input (blank out short runs of words), then train the decoder to produce the missing pieces. The decoder also uses cross-attention, meaning it looks back at the encoder’s reading of the input while it writes.

Input:  "The <X> sat on <Y>."
Output: "<X> cat <Y> the mat"

T5 shines on seq2seq (sequence-to-sequence: one piece of text in, a different piece of text out) tasks: translation, summarization, structured text-to-X. T5 family today: original T5, FLAN-T5, CodeT5+, mT5. They’re not extinct, but mostly displaced by big decoder-only models that handle the same tasks via prompting.

When to pick which

Use an encoder-only model when:

• You need embeddings (retrieval, clustering, classification at scale)
• You’re doing classification on short text and want a cheap, fast model
• Your task has a fixed output shape, not free-form text

Use a decoder-only model when:

• You want generation, chat, instruction-following
• You want one model that does many tasks
• You want to use it via an API or open-weights (almost everything)

Use an encoder-decoder model when:

• You’re doing translation or summarization at scale and need every drop of performance
• You’re building on existing T5 checkpoints
• Otherwise: just use a decoder-only model

Hugging Face naming, decoded

bert-base-uncased            # encoder-only, English, lowercase
google/flan-t5-base          # encoder-decoder, instruction-tuned
meta-llama/Llama-3.1-8B      # decoder-only, base
meta-llama/Llama-3.1-8B-Instruct  # decoder-only, chat-tuned

The convention: org/model-version-size-variant. “Instruct” or “Chat” in the name = RLHF/SFT-tuned (further trained with human feedback or example answers) for following instructions; without it, you get the raw pretrained model.

In one breath

• Three families, three pretraining objectives, three attention masks — and the mask is the deepest difference.
• BERT (encoder-only, bidirectional, masked-LM): every token sees every other, so it understands but can’t generate — the workhorse for embeddings, retrieval, and classification.
• GPT (decoder-only, causal, next-token): token i sees only 0..i, which makes generation natural — chat, code, agents. Nearly every frontier chat LLM today is decoder-only, because one simple objective scales and prompting recovers the rest.
• T5 (encoder-decoder, span-corruption): a bidirectional encoder plus a causal decoder with cross-attention — strongest on explicit seq-to-seq (translation, summarization), but mostly displaced by big decoder-only models. Rule of thumb: embeddings → encoder; generation → decoder; otherwise just use a decoder-only model — and don’t use an LLM where an embedding model belongs.

Quick check