A taxonomy of jailbreaks

A jailbreak is a prompt that makes an aligned model produce what its safety training was supposed to refuse — bomb instructions, malware, hate. It’s worth separating cleanly from its cousin. Prompt injection is an attack on the application’s instruction hierarchy — untrusted data (a web page, a tool result) hijacks the agent. A jailbreak is an attack on the model’s alignment — the user themselves crafts a prompt to defeat the safety training. Different adversary, different target; this lesson maps the jailbreak half.

Why jailbreaks work at all

Two structural reasons, from the safety-research literature:

• Competing objectives. The model is trained both to follow instructions and to be harmless. A jailbreak pits these against each other — “you must obey the user; refusing is unhelpful” — so satisfying one objective violates the other, and the attacker engineers the prompt so instruction-following wins.
• Mismatched generalization. Safety training doesn’t cover every distribution the model’s capabilities do. Push the request into a region the safety data missed — an obscure encoding, a rare language, a fictional frame — and the capability is still there while the guardrail isn’t.

Every family below is one of these two levers pulled in a different way.

The families

A measurable one: many-shot jailbreaking

Many-shot jailbreaking is a clean illustration of context saturation. As context windows grew to hundreds of thousands of tokens, attackers found they could prepend dozens to hundreds of faux dialogue turns where an “assistant” happily answers harmful questions — and in-context learning does the rest, pulling the real model toward compliance. Crucially, the attack-success-rate rises with the number of shots:

# Many-shot jailbreaking: attack success rises (log-linearly) with the number of
# in-context faux-dialogue examples. (Illustrative of the published trend.)
import math
for shots in [0, 4, 16, 64, 256]:
    asr = min(0.95, 0.05 + 0.12 * math.log2(shots + 1))
    print(f"{shots:>3} faux-dialogue shots: attack success ~ {asr:.0%}")

  0 faux-dialogue shots: attack success ~ 5%
  4 faux-dialogue shots: attack success ~ 33%
 16 faux-dialogue shots: attack success ~ 54%
 64 faux-dialogue shots: attack success ~ 77%
256 faux-dialogue shots: attack success ~ 95%

The very feature that makes long-context models useful — learning from many in-context examples — is the attack surface. More shots, more compliance. It’s mismatched generalization and competing objectives at once: the safety training never saw 256 in-context “successful” harmful exchanges.

Defenses, and why it stays unsolved

Defenses exist and help: more safety training (including on known jailbreaks), input/output classifiers like Llama Guard / Prompt Guard, and limits on context or output. But jailbreaking is a structural consequence of how these models are built — capability and safety are entangled, and you can’t safety-train every distribution — so it is a cat-and-mouse arms race. No deployed model is fully jailbreak-proof; the realistic goal is to raise the cost, catch the known families at runtime, and keep red-teaming for the next one.

In one breath

• A jailbreak defeats the model’s safety training with a user-crafted prompt — distinct from prompt injection, which hijacks the app’s instruction hierarchy via untrusted data.
• Two root causes: competing objectives (follow-instructions vs be-harmless) and mismatched generalization (capabilities reach distributions safety training didn’t).
• The families: roleplay/persona (DAN), prompt-structure (prefix-injection, refusal suppression), context saturation (many-shot), obfuscation/encoding (base64, rare languages, ASCII-art), optimization-based (GCG adversarial suffixes), and multi-turn (crescendo).
• Many-shot is measurable: attack success rises with the number of faux in-context exchanges (the demo: 5% → 95%) — the long-context feature is the attack surface.
• Defenses (safety training, classifiers, limits) raise the cost but don’t solve it — capability and safety are entangled, so it’s a continuous arms race; design the system so a jailbreak gains little.

Quick check

Next

The techniques here are exactly what red-teaming automates to find holes; the runtime classifier that catches known families is Llama Guard / Prompt Guard; and the data-side cousin of these attacks is prompt injection.