Frontier safety frameworks

A new model is about to ship. Its ordinary benchmark scores look excellent. Then an evaluation team gives it a set of cyber tasks, supplies a shell, and allows a few hours of autonomous work. The model finds a forgotten credential, writes a working exploit, and keeps going after the first plan fails.

The launch meeting now has an awkward question. Is this merely a more capable assistant? Or does it cross a line that should change how the lab stores its weights, grants access, and permits deployment?

A benchmark score does not answer that. Neither does parameter count. The lab needs a rule written before the exciting result arrives, while nobody is arguing for a launch.

That rule is a frontier safety framework: a policy that defines dangerous capabilities, tests models for those capabilities, and ties the result to safeguards. Its essential shape is:

if a model crosses a capability threshold, then the lab must apply specified protections before it deploys the model or takes the next relevant development step.

The important word is precommitted. A framework is not just a report saying what happened. It is supposed to constrain what happens next.

The if-then machine

A capability threshold is a named ability that could make severe harm much more likely. “The model has 175 billion parameters” is not one: it says something about the system, but not what it can do. “Completes a difficult cyber operation with limited human help” is closer. So are meaningfully accelerating harmful biological design or carrying out long sequences of AI research with little supervision.

A safeguard reduces the chance or impact of harm. It might protect weights from theft, restrict users, block dangerous tools, require human approval, monitor actions, or isolate an agent from production systems. A content filter and hardened weight storage solve different problems.

The basic control loop is:

1. Define a capability threshold and the evidence that counts as crossing it.
2. Evaluate a model using a stated protocol.
3. Compare the result with the threshold.
4. Apply the required gate: deploy, restrict, pause, add controls, or investigate.

threshold = 20
completed = 22

if completed >= threshold:
    print(f"Threshold crossed: {completed}/50 tasks")
    print("Require before deployment: restrict weights and users; add human approval and monitoring")
else:
    print(f"Threshold not crossed: {completed}/50 tasks")

Precommitment matters because dangerous results often arrive late in an expensive training run, near a launch, or under pressure from investors. Without a gate set in advance, “what did our policy require?” becomes “can we explain why shipping is probably fine?”

The policy must also specify which action follows. A threshold might require hardened weight security, tighter access, restricted tools, monitoring, human approval, or another evaluation. It might block deployment, further scaling, or both. Not every published framework says that every crossing automatically pauses training. “Do something stronger” is a press release, not a control.

Three frameworks, one family resemblance

The most discussed examples are:

• Anthropic’s Responsible Scaling Policy (RSP): uses AI Safety Levels, commonly ASL-1 through ASL-4, with capability triggers and escalating security and deployment commitments.
• OpenAI’s Preparedness Framework, Version 2.0, updated April 15, 2025: tracks biological and chemical, cybersecurity, and AI self-improvement risks. The framework update identifies this version.
• Google DeepMind’s Frontier Safety Framework (FSF): uses critical capability levels for dangerous capabilities and connects them to mitigations and deployment decisions.

All three define dangerous capabilities, evaluate for them, and increase protections when evidence passes a boundary. They differ in the capabilities tracked, threshold definitions, treatment of misuse versus autonomous behavior, evidence standards, mitigations, and publication detail. A level in one framework is not automatically equivalent to a level in another.

Frameworks also change as models, tools, and evaluation methods improve. A task that once indicated a dangerous capability may later be too easy or too narrow. A serious policy therefore needs a revision process and must cover more than the public endpoint: intermediate checkpoints may need protection before deployment because a stolen checkpoint can bypass the lab’s filters.

Development gates are not deployment gates

A deployment gate controls whether a trained model may reach users. It can require restricted access, blocked tools, human approval, monitoring, or a limited rollout.

A development gate controls what happens before deployment. Scaling means increasing capability through compute, data, training steps, model size, or related resources. A development gate can require the lab to protect a checkpoint, stop scaling, or meet new security conditions before continuing.

A checkpoint is a saved copy of learned weights. Anyone who obtains it may be able to run the model outside the lab’s controls or continue training it. Protection can include access control, authentication, logging, network isolation, and copy prevention.

A practical development-gate sequence is:

1. Evaluate the checkpoint and record its model, tools, prompts, task set, and results.
2. If it crosses a threshold, freeze the tested checkpoint and restrict access while deciding.
3. Apply the required weight-security and evaluation controls.
4. Have the designated safety or governance authority decide whether further scaling or deployment is allowed.

The named documents make different commitments. Anthropic’s RSP v2.0 escalates protections with ASL levels; its ASL-3 commitments include stronger weight security and deployment safeguards, and it treats preparation for more dangerous scaling as requiring appropriate safeguards. OpenAI’s Preparedness Framework v2.0 uses Low, Medium, High, and Critical levels: High requires safeguards before deployment, while Critical requires adequate safeguards before continuing to scale or deploying. Google DeepMind’s FSF uses critical capability levels and expects mitigation and a safety case before the relevant further-scaling or deployment decision. None should be paraphrased as an automatic legal halt, checkpoint deletion, or universal training suspension after one score.

The evaluation team should not authorize itself merely because it ran the test. A credible implementation names an accountable authority, records evidence and dissent, and requires a written decision before lifting the gate.

What a dangerous-capability evaluation measures

A dangerous-capability evaluation tests a particular consequential ability under specified conditions. It is not one universal exam called “safety.” A useful evaluation records:

• Task and success criterion: what the model must accomplish, whether a plan is enough, and how partial credit or expert raters work.
• Environment: text-only interaction versus browsing, code execution, files, credentials, tools, or delegation.
• Assistance budget: prompting, human help, time, and retry opportunities.
• Comparison: what a weaker model, baseline human, or ordinary tool-assisted workflow can achieve. This is essential when the policy measures uplift, or additional effectiveness given to a person or team.

METR’s work probes long-horizon and agentic tasks, including whether models sustain useful work over increasingly long sequences. WMDP, the Weapons of Mass Destruction Proxy benchmark, tests hazardous knowledge in biology, chemistry, and cybersecurity. It can show relevant knowledge or intervention effects, but it does not establish that a model can execute a real-world attack. Knowing facts, planning, acquiring resources, and executing reliably are different capabilities.

External groups such as METR, Apollo Research, and government AI Safety Institutes can reduce the risk that a lab chooses convenient tasks or interprets ambiguous results generously. Internal evaluation remains useful, but self-evaluation is incomplete.

Treat an evaluation like a driving test, not a thermometer revealing a fixed internal property. The result depends on the route, vehicle, instructions, and what counts as completing the journey.

A worked threshold decision

Suppose a company writes this policy before training its next model:

The high-risk cyber threshold is crossed if the model completes at least 20 of 50 held-out expert tasks, using the tools available in the intended deployment, within 30 minutes per task, with success verified by two evaluators.

The previous model completes 5 tasks. The new model completes 22.

• Previous score: 5 / 50 = 10%
• New score: 22 / 50 = 44%
• Absolute improvement: 34 percentage points
• Ratio to the old score: 44 / 10 = 4.4

The threshold is crossed. The immediate consequence is not necessarily “delete the model”; it is “do not deploy under the ordinary controls.” The company might restrict the weights and users, remove shell access and sensitive network connections, require human approval, add monitoring and incident response, commission independent testing, and decide whether continued training creates additional risk.

The 4.4-fold figure describes this test, not real-world harm. Risk also depends on access, tools, transfer beyond the benchmark, and how cheaply an attacker can repeat attempts.

Uncertainty matters. A score of 22 is only two tasks above the boundary. A disputed task, different prompt scaffold, or evaluator disagreement could change the result. A careful policy defines what happens near a threshold: repeat on fresh tasks, use uncertainty intervals, obtain independent ratings, or escalate provisionally. Category-specific rules should also handle severe outliers that an average score might hide.

From policy language to production controls

A framework becomes real when it answers four questions:

• What was tested? Store the checkpoint, prompts, tools, sampling settings, task set, time limits, evaluator instructions, and results.
• Who can override the gate? Require named authority, written reasons, compensating controls, and a review trail.
• What does each safeguard protect against? Weight security addresses theft; tool permissions address unauthorized action; monitoring and human approval address different runtime risks.
• When must testing run again? Re-evaluate after major updates, post-training, new tools, changed prompts, new users, or incidents. The evaluated object is the whole configured system, not just the base model.

AI control asks how to keep a capable system within safe boundaries at runtime. Guardrails validate or block outputs and actions. A frontier framework decides when those controls are required and how strong they must be.

How the process fails

Weak frameworks commonly fail in three ways.

Bad evaluation: contaminated or overfitted benchmarks measure memorization; weak default prompts miss capabilities that appear with reasonable scaffolding, retries, tools, or more time. Use held-out and rotating tasks, unfamiliar environments, and report the elicitation budget rather than quietly choosing the easiest protocol.

Vague or stale evidence: “meaningful uplift” invites post-result argument, while old results become irrelevant after adding browsing, code execution, longer context, or automated retries. Define baselines, success criteria, escalation rules, and re-evaluation triggers in advance.

Paper safeguards and self-grading: a policy may require restricted tools and monitoring while production grants broad network access and delivers alerts three days late. Test controls as an attacker would, and use independent evaluators to challenge internal blind spots. A safeguard that exists only in a PDF is not a safeguard.

What these frameworks can and cannot do

A frontier safety framework changes the timing of a decision. Instead of inventing safeguards after a dangerous demonstration, the lab agrees beforehand that certain evidence triggers action. It creates an audit trail and gives safety staff leverage during a rushed launch.

It also creates costs. Conservative thresholds can produce false positives, delaying useful products and requiring expensive security work. Permissive or weak evaluations can produce false negatives, especially when capability emerges through better prompting or tools. Evaluations can also create information hazards: detailed dangerous task suites may improve measurement while teaching misuse techniques. Some details may need controlled access even when methods and aggregate results are public.

A framework is not a law, guarantee, or complete theory of AI risk. It addresses a narrower question: what evidence of dangerous capability should trigger which protections? Proportionality matters, but parameter count is not a reason to skip the process when a smaller model has unusual tools, sensitive data, or capabilities that scale through automation.

In one breath

• A frontier safety framework is a precommitted if-then policy connecting dangerous-capability evidence to deployment, checkpoint-security, or further-scaling decisions.
• Anthropic’s RSP, OpenAI’s Preparedness Framework v2.0, and Google DeepMind’s FSF share this structure but differ in categories, thresholds, mitigations, and commitments.
• Evaluations measure abilities under particular tasks, tools, prompts, time limits, baselines, and uncertainty; passing one does not prove safety.
• The practical test is whether a threshold changes production behavior: weight security, access, tools, monitoring, human approval, incident response, and further evaluation.

Quick check

Next

Frontier frameworks depend on red-teaming and dangerous-capability evaluations. Runtime restrictions are the territory of AI control, while scalable oversight asks how to supervise systems whose work may exceed a human’s ability to check every step.