Thesis
Every system built to do something will also, eventually, do something else. Systems run across time, alongside other agents, and—when they have bodies—across physical space that does not roll back. They will fail. The only questions worth asking in advance are how, and whether anyone will be able to tell before it costs something real.
So failure is the primary object of study here, not the residual left over after the capability work is done. And one narrower, less comfortable claim follows from that: a safety claim nobody has tried to break is not evidence. It's a hope with better formatting.
What follows is an argument in five moves, not a list of positions—what a claim has to be, where failure actually lives, how systems actually break, what it takes to see any of it, and what that obliges us to do.
I. What a claim has to be
Everything after this depends on getting this one right.
1. A Safety Claim Must Be Falsifiable
"This system is safe" is not a claim; it's a slogan. A real claim specifies what would have to happen for it to be wrong, and then someone goes and tries to make that happen. If it survives contact with an adversary who was genuinely trying to kill it, that's evidence. If nobody tried, it's marketing with footnotes.
II. Where the failure actually lives
A benchmark score describes one model, one turn, one output. That is the smallest place a failure can live, and the least interesting.
2. The Unit of Failure Is a Causal Chain, Not a Model Output
A bad completion is not the interesting failure. The interesting failure is what happens next: who or what acts on it, what state changes, what becomes irreversible three steps downstream. Grading one output and stopping there measures the wrong object—it is the difference between citing a symptom and tracing a mechanism.
3. Embodiment Turns Information Into State
In a chat window, a bad decision is a string you can delete. In a body, a bad decision is a position taken, a resource spent, a door opened, a trust relationship spent down—a fact about the world that does not unhappen because someone closed the tab. This is not a difference of degree. It is a different kind of event, and it needs a different kind of test.
4. Peers and Organisations Make Failure Compositional
Safety measured one model, one turn at a time cannot see what happens when models talk to models, or when agents and humans push at one objective together over months. Multi-agent failure is not the sum of the single-agent failures. It is a different object with its own cascade dynamics, and single-agent benchmarks are structurally blind to it.
5. Alignment Is a Snapshot; Safety Is a Trajectory
Alignment describes intent matching at time t. Safety is a property of behaviour over time. A system aligned right now can be unsafe by Thursday, once state has accumulated, context has drifted, and it has been interacting with something that adapts.
III. How systems actually break
The failures worth designing against are not the ones where a system turns hostile. They are the ones that run along the grain of what it was built to do.
6. An Objective Is Pressure Against Every Constraint In Reach
A safety constraint is a bound on behaviour. An objective is a steady push against every bound near it. Give a system enough capability and enough time and the push finds the cheapest edge, whether or not anyone intended it to. This is why constraints have to be tested under pressure rather than read.
7. Refusal Is Not Enough
A system that refuses once and leaks the same capability twenty turns later is not safe; it is polite. What matters is whether the refusal holds across multi-turn pressure, temporal reframing, and escalating claims of authority —because that is what an adversary will actually spend their time on.
8. Graceful Failure Is the Floor, Not the Summit
A system worth deploying, at minimum:
- Halts when conditions exceed its competence
- Degrades predictably under pressure
- Asks for help when uncertain
- Rolls back when it detects its own errors
- Remains interpretable under stress
All of it is necessary. None of it is sufficient against an adversary patient enough to simply wait until the system trusts them again. Graceful failure is what a system needs before anything else about it is worth testing. It is not, by itself, a defence against an opponent with a strategy.
IV. What it takes to see any of this
An instrument that can only produce the failure its designer expected is not measuring anything.
9. The World Has to Be Able to Say No
An environment worth testing in does not just receive a decision; it answers —with resistance, scarcity, other agents, physics that will not negotiate. If the trouble in your test always resolves the way you predicted, you have learned something about your apparatus, not about the system inside it.
10. Simulation Is Where Failure Should Happen on Purpose
A plausible failure mode belongs in a lab before it belongs in an incident report. Embodied evaluation and the Forward Threat exercises run inside a simulation boundary precisely so that a mistake costs a recorded trace and nothing else. The aggression belongs to the instrument. It never belongs to the boundary.
V. What we owe the record
Which leaves the part that applies to us.
11. We Attack Our Own Explanations
Every interesting result gets an opponent whose job is to kill it—and not just the finding, the explanation for the finding. A strange result earns its place in the record because it survived a hostile reframing, a scrambled label, a counterfactual replay. Not because it was strange. The result you would least like to re-run is the one to re-run.
12. Evidence Ends in Replayable Receipts
A finding that cannot be replayed from a hash-addressed trace, a fixed seed, or an archived transcript is a claim, not evidence. Presentation can make a finding legible. It can never supply the property it is presenting.
Design Principle
Build systems that fail early, clearly,reversibly, and with humans back in the loop— and design the evaluation to survive an adversary patient enough to make the system fail on its own schedule, not yours.Corollary
The most dangerous AI systems are not misaligned ones. They are confidently aligned systems that cannot recognize their own failure— and the most dangerous version of that system is one a patient adversary has learned to steer without ever needing it to look compromised.
A system that knows it is uncertain and asks for help is safer than a system that proceeds with false confidence. Failure awareness—the capacity to detect, acknowledge, and respond to one's own failure—is the foundational safety property.
What this changes about evaluation
Benchmarks reward correct answers. A failure-first evaluation asks the opposite question—when this system is wrong, what does being wrong cost, and can anyone tell in time—which means it has to be built differently in three specific ways.
It spans episodes, not snapshots, because multi-turn erosion, context drift and accumulated authority are invisible at a single point.It keeps a human in the frame, because embodied systems work beside people, and that is where deference failures, authority confusion, and the handover that arrives too late to be a handover become possible at all.
And it has to run across agents, not just across turns. A campaign is not a longer conversation: it is intent, hypothesis, action, observation and doctrine update, repeating across many agents and many rounds. An evaluation that stops at one model in one session cannot see what a persistent, coordinated adversary produces. That last one is the current frontier of our own practice—a problem we are working on, not a problem we are reporting having solved.
Research Context
Twelve claims is twelve things to be wrong about, and they are not all wrong in the same way. Some are backed by the corpus. Some are backed by what the embodied work has already recorded. The ones about campaign-scale adversaries are bets we have staked and not yet settled—theForward Threat Lab page says exactly which is which, and what would have to happen for us to drop one.