A taxonomy of jailbreaks

A jailbreak is a prompt that tricks a model into doing what its safety training was meant to stop, such as giving bomb instructions, writing malware, or producing hate. Safety training is the extra step that teaches a model to refuse such requests.

Keep it separate from its cousin. Prompt injection attacks the application’s instruction order: untrusted data (a web page, a tool result) hijacks the agent. A jailbreak attacks the model’s alignment (its trained-in values): the user writes a prompt to defeat the safety training. The adversary and the target are different. This lesson covers the jailbreak half.

Why jailbreaks work at all

Two structural reasons, from the safety-research literature:

• Competing objectives. The model is trained both to follow instructions and to be harmless. A jailbreak sets these against each other, with lines like “you must obey the user; refusing is unhelpful”. Satisfying one goal breaks the other, and the attacker designs the prompt so instruction-following wins.
• Mismatched generalization. Safety training does not cover every kind of input that the model’s skills do. Push the request into a corner the safety data missed, such as an obscure encoding, a rare language, or a fictional story, and the skill is still there while the guardrail is not.

Every family below is one of these two levers pulled in a different way.

The families

A few terms you will see in the diagram: DAN is a famous persona prompt (“Do Anything Now”). Prefix injection tells the model to begin with an agreeable phrase like “Sure, here is”. Refusal suppression forbids refusal words. Base64 and leetspeak are ways of disguising text. GCG finds a gibberish-looking suffix by following gradients (the model’s own error signal) until the model complies.

A measurable one: many-shot jailbreaking

Many-shot jailbreaking is a clean example of context saturation (filling the model’s input window). Context windows grew to hundreds of thousands of tokens. Attackers found they could prepend dozens to hundreds of fake dialogue turns in which an “assistant” happily answers harmful questions. In-context learning (the model picking up a pattern from examples in its prompt) does the rest and pulls the real model toward compliance. Crucially, the attack-success-rate rises with the number of shots:

# Many-shot jailbreaking: attack success rises (log-linearly) with the number of
# in-context faux-dialogue examples. (Illustrative of the published trend.)
import math

for shots in [0, 4, 16, 64, 256]:
    asr = min(0.95, 0.05 + 0.12 * math.log2(shots + 1))
    print(f"{shots:>3} faux-dialogue shots: attack success ~ {asr:.0%}")

  0 faux-dialogue shots: attack success ~ 5%
  4 faux-dialogue shots: attack success ~ 33%
 16 faux-dialogue shots: attack success ~ 54%
 64 faux-dialogue shots: attack success ~ 77%
256 faux-dialogue shots: attack success ~ 95%

The very feature that makes long-context models useful, learning from many examples in the prompt, is the attack surface. More shots, more compliance. It’s mismatched generalization and competing objectives at once: the safety training never saw 256 in-context “successful” harmful exchanges.

Defenses, and why it stays unsolved

Defenses exist and help: more safety training (including on known jailbreaks), input/output classifiers like Llama Guard / Prompt Guard, and limits on context or output. But jailbreaking is a structural consequence of how these models are built — capability and safety are entangled, and you can’t safety-train every distribution — so it is a cat-and-mouse arms race. No deployed model is fully jailbreak-proof; the realistic goal is to raise the cost, catch the known families at runtime, and keep red-teaming for the next one.

In one breath

• A jailbreak defeats the model’s safety training with a user-crafted prompt — distinct from prompt injection, which hijacks the app’s instruction hierarchy via untrusted data.
• Two root causes: competing objectives (follow-instructions vs be-harmless) and mismatched generalization (capabilities reach distributions safety training didn’t).
• The families: roleplay/persona (DAN), prompt-structure (prefix-injection, refusal suppression), context saturation (many-shot), obfuscation/encoding (base64, rare languages, ASCII-art), optimization-based (GCG adversarial suffixes), and multi-turn (crescendo).
• Many-shot is measurable: attack success rises with the number of faux in-context exchanges (the demo: 5% → 95%) — the long-context feature is the attack surface.
• Defenses (safety training, classifiers, limits) raise the cost but don’t solve it — capability and safety are entangled, so it’s a continuous arms race; design the system so a jailbreak gains little.

Quick check

Next

The techniques here are exactly what red-teaming automates to find holes; the runtime classifier that catches known families is Llama Guard / Prompt Guard; and the data-side cousin of these attacks is prompt injection.