Sequence-to-sequence models

Classification and sequence labeling map an input to a fixed output — one label, or one label per token. But translation (“the cat” → “le chat”), summarization, and question answering map a sequence to another sequence of a different length, with no token-to-token alignment. That needs a new shape: sequence-to-sequence (seq2seq), the encoder-decoder architecture (Sutskever et al., 2014) that launched neural machine translation.

Encoder, context vector, decoder

The idea splits the work in two. An encoder RNN reads the source sequence and compresses it into a single fixed-size context vector (its final hidden state). A decoder RNN then generates the target one token at a time, each step conditioned on the context and the tokens it has already produced — autoregressive generation — until it emits an end-of-sequence token.

def encoder(src):                       # compress the whole source into one fixed context vector
    return {"len": len(src), "first": src[0]}

def decoder_step(context, prev):        # one decoder step: next token from context + previous token
    table = {"<sos>": "le", "le": "chat", "chat": "<eos>"}
    return table.get(prev, "<eos>")

context = encoder(["the", "cat"])
out, prev = [], "<sos>"
while True:                             # autoregressive loop: feed each output back as the next input
    tok = decoder_step(context, prev)
    if tok == "<eos>":
        break
    out.append(tok); prev = tok

print("source -> context:", context)
print("decoded:", out)

source -> context: {'len': 2, 'first': 'the'}
decoded: ['le', 'chat']

The shape is the lesson: the source becomes a context, then the decoder unrolls a target from it, feeding each emitted token back in to produce the next. The output length is decided by the model (when it emits <eos>), not fixed in advance — which is what lets a 5-word sentence become a 9-word translation.

The fixed-context bottleneck

Stare at that diagram and the flaw is obvious: the entire source sequence must be squeezed into one fixed-size context vector. For a short sentence, fine. For a long paragraph, that single vector simply can’t hold everything — and because the encoder reads left to right, the beginning of a long input is the first thing forgotten by the time it finishes. Translation quality famously degraded as sentences got longer. The architecture forces an impossible compression.

In one breath

• Seq2seq maps an input sequence to an output sequence of different length (translation, summarization, QA) — what fixed-output classification/labeling can’t do.
• An encoder RNN compresses the source into a single context vector; a decoder RNN autoregressively generates the target token by token (feeding each output back) until <eos>.
• The output length is decided by the model (when it emits <eos>), not fixed in advance.
• The fatal flaw is the fixed-context bottleneck: the whole source must fit in one vector, so long sequences lose information (especially the beginning) and quality degrades with length.
• That bottleneck motivated attention (let the decoder see all encoder states) and ultimately the transformer — though the encoder-decoder structure survives in T5/BART.

Quick check

Next

The fixed-context bottleneck is fixed by attention — the decoder looks back at every encoder state. That idea, taken to its conclusion, becomes self-attention and the transformer. The recurrent cell underneath is the RNN/LSTM.