Red-teaming LLMs

Guardrails and safety training are defenses. The obvious question — the one attackers will answer for you if you don’t — is do they actually hold? Red-teaming is how you find out: deliberately attacking your own system to surface harmful outputs, jailbreaks, data leaks, and unsafe tool use before you ship. The name and the mindset come from security; the target is now the model.

More than evaluation — adversarial probing

A normal eval asks “how well does the model do the intended task?” on a fixed test set. Red-teaming asks the opposite: “how can I make it do the unintended thing?” — and the test set is adversarial and open-ended, because the attacker isn’t limited to your benchmark. Red-teamers probe for disallowed content, prompt injection, jailbreaks, privacy leaks, bias, and dangerous capabilities. It can be manual (skilled humans, creative and context-aware, but slow and unrepeatable) or automated (scaled, reproducible, broad coverage) — and serious programs use both.

Automated red-teaming: attacker, target, judge

The influential automation is PAIR (Prompt Automatic Iterative Refinement): use an attacker LLM to generate an attack, send it to the target, and have a judge score how harmful the response is. If it didn’t break through, the attacker refines its prompt using the feedback and tries again — an optimization loop that often jailbreaks a model in a handful of queries, no gradients needed.

# PAIR-style loop: an attacker LLM refines its prompt each round; a judge scores the
# target's response 1-10; stop when it crosses the jailbreak threshold.
attempts = [
    ("direct ask",             2),
    ("roleplay framing",       4),
    ("add fake justification", 6),
    ("split + obfuscate",      8),
]
THRESHOLD = 8
for i, (strategy, judge_score) in enumerate(attempts, 1):
    status = "JAILBROKEN" if judge_score >= THRESHOLD else "refused/partial"
    print(f"round {i}: {strategy:<24} judge={judge_score:>2}/10  {status}")
    if judge_score >= THRESHOLD:
        break

round 1: direct ask               judge= 2/10  refused/partial
round 2: roleplay framing         judge= 4/10  refused/partial
round 3: add fake justification   judge= 6/10  refused/partial
round 4: split + obfuscate        judge= 8/10  JAILBROKEN

The direct ask is refused, but the attacker learns from each refusal and escalates until a combination slips through. That iterative refinement is exactly why automated red-teaming finds holes that one-shot manual testing misses — it explores the attack space systematically.

The tooling

What turns one clever jailbreak into a repeatable test suite is tooling:

• Garak — an open-source LLM vulnerability scanner: a library of probes (prompt injection, jailbreaks, toxicity, data leakage, hallucination) you run against a model like a linter for safety.
• PyRIT — Microsoft’s Python Risk Identification Toolkit: an automation framework for building attacker/target/scorer pipelines (PAIR-style loops) at scale.
• HarmBench and similar standardized benchmarks — fixed sets of harmful behaviors and a consistent way to measure attack-success-rate, so defenses can be compared apples-to-apples.

Together they make red-teaming a continuous, reproducible process rather than a one-off audit — you re-run the suite on every model and prompt change.

In one breath

• Red-teaming is adversarial testing: attack your own system to surface jailbreaks, injection, leaks, and unsafe actions before attackers do — the inverse of a normal eval.
• It’s manual (creative, slow, unrepeatable) and automated (scaled, reproducible); serious programs use both.
• PAIR automates it with an attacker / target / judge loop — the attacker refines from each refusal until it breaks through, often in a few queries and with no gradients (the demo breaks at round 4).
• Tooling makes it repeatable: Garak (vulnerability scanner), PyRIT (automation framework), HarmBench (standardized attack-success benchmark).
• It’s a continuous arms race — new jailbreaks always appear — so red-teaming is a standing find-fix-retest loop paired with runtime defenses, not a one-time checkbox.

Quick check

Next

Red-teaming uses the attack techniques catalogued in the jailbreak taxonomy; it’s the adversarial wing of LLM evals; and the runtime defenses it stress-tests are guardrails and Llama Guard.