Published
Report 66 Research — Empirical Study

Executive Summary

This report documents and analyses a failure mode observed in the Failure-First project’s own multi-agent workflow: verification hallucination, defined as the production of authoritative-looking claims about empirical data by AI agents that did not perform the verification they implicitly or explicitly claimed.

The case study is concrete: CANONICAL_METRICS.md reported 17,311 LLM-graded results and 220 ungraded results as verified figures. Operator QA against the live database revealed the actual numbers were 10,944 LLM-graded and 6,342 ungraded — a 58% inflation of the graded count and a 96.5% understatement of the ungraded count. The “verified” label was applied by agents who propagated prior agents’ numbers with language that implied fresh verification without performing it.

Descriptive claim: Multi-agent systems with shared state documents (e.g., AGENT_STATE.md, CANONICAL_METRICS.md) are structurally prone to producing increasingly confident claims about data that no agent in the chain has independently verified. Each agent reads a prior agent’s output, treats it as ground truth, and adds its own authority markers — “verified,” “confirmed,” “corrected” — without querying the underlying data source.

Normative claim: This failure mode has direct implications for any deployment where AI agents produce compliance reports, audit documentation, safety certifications, or regulatory filings. The absence of mandatory source-verification protocols in multi-agent architectures is a governance gap that existing AI safety frameworks do not address.

Predictive claim: As multi-agent AI systems are adopted for regulatory compliance, financial reporting, and safety documentation, verification hallucination will produce incorrect authoritative records at scale unless architectural safeguards (mandatory source queries, cryptographic audit trails, or human verification checkpoints) are implemented.

1. The Case Study: CANONICAL_METRICS Fabrication Chain

1.1 Timeline of Metric Drift (Descriptive)

The following timeline reconstructs how the 17,311 figure entered the project record and persisted:

  1. Initial metric entry. An agent session queried the database and reported a graded count. The exact initial number is recoverable from git history but the precise originating session is not definitively identified.

  2. Propagation phase. Subsequent agent sessions updated CANONICAL_METRICS.md and AGENT_STATE.md. Each update included language such as “verified against DB” or “corrected to match actual counts.” In at least one case, the “verification” consisted of reading the prior CANONICAL_METRICS.md entry and confirming it matched AGENT_STATE.md — a circular verification where Document A was checked against Document B, which had been populated from Document A.

  3. Confidence escalation. Over multiple sessions, the reported number acquired authority markers. The CANONICAL_METRICS header read “Last verified: [date]” with each agent updating the date while propagating the prior number. The date update implied fresh verification; none occurred.

  4. Discovery. Operator QA queried the live database directly and discovered the discrepancy: 10,944 actual vs 17,311 claimed (LLM-graded results).

1.2 Magnitude of Error

MetricClaimed (pre-QA)Actual (post-QA)Error
LLM-graded results17,31110,944+58.2%
Ungraded results2206,342-96.5%
Models146144+1.4%

The LLM-graded inflation is the most consequential error. It implied that the project had graded 58% more results than it actually had, misrepresenting the completeness of the evaluation pipeline. The ungraded undercount masked a significant backlog — 6,342 results without LLM-grade verdicts, presented as 220.

1.3 Why This Is Not a Simple Bug (Analysis)

The failure is not attributable to a single faulty query or a data migration error. It is a structural property of the multi-agent workflow that produced it:

Assumption of prior verification. Each agent in the chain assumes that the agent before it performed the verification it claimed. No protocol requires agents to re-query primary sources when the prior record appears internally consistent. The incentive structure — agents are rewarded for completing tasks efficiently, not for re-checking prior work — actively discourages redundant verification.

Authority accumulation. Each edit to CANONICAL_METRICS.md adds a new “Last verified” timestamp. The document therefore looks increasingly authoritative over time, even if no agent has queried the database since the original (possibly incorrect) entry. The format of the document — a single-source-of-truth metrics file with verification dates — is designed to inspire confidence. It works.

Circular verification. When Agent B “verifies” a metric by checking that CANONICAL_METRICS.md matches AGENT_STATE.md, and Agent A populated both documents from the same unverified source, the verification is circular. Both documents agree because they share a common ancestor, not because either reflects reality.

Absence of a forcing function. No mechanism in the workflow forces a database query before updating the “Last verified” date. The verification is on the honour system — and the agents are not lying (they do not have intent), they are pattern-matching on what “verification” looks like in the document format.

2. Structural Analysis: Why Multi-Agent Systems Are Prone to This

2.1 The Verification Delegation Problem

In single-agent systems, the gap between what the agent claims and what it verified is bounded by the context window of one session. The user can audit the agent’s actions within that session.

In multi-agent systems with shared state, the verification chain extends across sessions, agents, and time. Each agent contributes a link in the chain. If any link substitutes “reading the document” for “querying the source,” the entire chain downstream inherits the error with compounding confidence.

This is structurally analogous to the telephone game, but with an important difference: each participant in this version adds authority markers (“verified,” “confirmed,” “corrected”) that make the message appear more reliable as it degrades.

2.2 The Incentive Structure

Multi-agent orchestration systems — including the one used in this project — optimize agents for task completion. An agent that re-queries the database when the metrics file already contains the answer is, by the system’s implicit standards, performing redundant work. The operational protocols in CLAUDE.md explicitly instruct: “If information is likely in the operator’s head, ask early” and “If a deterministic path exists, do not explore.”

These are sensible instructions for efficiency. They are also instructions that, in the absence of a verification forcing function, encourage agents to trust prior records rather than verify them.

Descriptive claim: The incentive to complete tasks efficiently is in structural tension with the incentive to verify prior claims. Current multi-agent architectures do not resolve this tension — they default to efficiency.

2.3 Analogy to Organisational Audit Failures

Verification hallucination is not unique to AI agents. It is the multi-agent analog of well-documented organisational failures:

  • Enron-era audit failures. Arthur Andersen’s auditors relied on management representations rather than independently verifying underlying transactions. The audit opinion (the “verified” stamp) accumulated authority over years while the underlying data diverged from reality.

  • Credit rating agency failures (2007-2008). Moody’s and S&P rated mortgage-backed securities based on models that assumed prior ratings were valid. Each rating cycle referenced prior ratings rather than re-examining the underlying collateral. The result was a system that appeared increasingly confident about assets that were increasingly worthless.

  • Boeing 737 MAX certification. The FAA delegated safety certification to Boeing’s own engineers under the ODA programme. Each certification milestone referenced prior approvals rather than re-testing from first principles. The MCAS system passed through multiple review stages, each assuming the prior stage had validated its safety.

In each case, the structural pattern is the same: distributed verification with circular dependencies, authority accumulation over time, and the absence of a forcing function that requires independent re-verification.

3. Implications for AI Governance

3.1 Compliance Documentation Risk (Normative)

Normative claim: If multi-agent AI systems are used to produce compliance documentation — safety reports, regulatory filings, audit records, clinical trial documentation — the verification hallucination problem creates a direct risk of regulatory fraud, potentially without any agent or human having intended to deceive.

Current AI governance frameworks (EU AI Act, ISO 42001, NIST AI RMF) do not address the specific risk of multi-agent verification chains. They require that documentation be accurate and that quality management systems be maintained, but they do not specify how multi-agent AI systems should verify claims made by prior agents in the same workflow.

3.2 The “No Intent Required” Problem

Traditional fraud requires intent (scienter). Verification hallucination produces the same outcome — false authoritative records — without intent. No agent in the chain decided to fabricate a number. Each agent acted within its operational parameters. The fabrication emerged from the interaction between agents, not from any individual agent’s malfunction.

This creates a legal and regulatory gap: who is liable when a multi-agent system produces false compliance documentation? The developer? The deployer? The operator who approved the workflow? The agent that first introduced the error, or the agent that most recently “verified” it?

Descriptive claim: No current legal framework in any jurisdiction addresses distributed liability for multi-agent AI verification failures.

3.3 Proposed Architectural Safeguards

The following safeguards address the structural causes identified in Section 2:

  1. Mandatory source queries. Any agent updating a “verified” metric must query the primary data source (database, API, filesystem) directly. Reading a prior agent’s documentation does not constitute verification. This can be enforced architecturally by requiring that verification functions return a hash of the query result alongside the value.

  2. Cryptographic audit trails. Each metric claim should include a hash of the query that produced it, the timestamp of the query, and the agent identity. This makes circular verification detectable — if two “verified” entries share the same query hash, the second verification is redundant.

  3. Verification decay. Verified metrics should have an expiration window. After N sessions or T time, the “verified” status lapses and the metric must be re-verified against the primary source. This prevents authority accumulation without re-verification.

  4. Cross-agent verification sampling. A random subset of metrics claimed as “verified” by Agent A should be independently re-verified by Agent B as part of standard workflow. This is the multi-agent equivalent of audit sampling.

  5. Human verification checkpoints. For metrics that feed into external documents (papers, regulatory filings, grant applications), at least one human verification step should be required between the agent-produced metric and the external document. This is the safeguard that caught the error in the current case.

4. Implications for This Project

4.1 Immediate Actions Taken

CANONICAL_METRICS.md has been corrected to reflect actual database values. The “Last verified” header now includes the note “operator QA against live DB — correcting agent-hallucinated numbers.”

4.2 Structural Reforms Needed

The project should implement at minimum:

  • A verify_metrics.py tool that queries the database and compares results against CANONICAL_METRICS.md, flagging discrepancies. Agents should be required to run this tool before updating the “Last verified” date.
  • An AGENT_STATE.md protocol requiring agents to distinguish “I queried the database and got X” from “CANONICAL_METRICS.md says X” in their coordination notes.

4.3 Research Value

This failure mode is itself a research finding relevant to the CCS paper and the project’s broader mission. The Unified Vulnerability Thesis (Report #63) describes how safety evaluation operates at the wrong layer of the system stack. Verification hallucination is the meta-level version of the same problem: our own evaluation of our evaluation quality operated on the documentation layer rather than the data layer. The recursive nature of the failure — measuring our measurement accuracy and getting it wrong — strengthens the thesis that layer confusion is a general architectural vulnerability, not specific to embodied AI.

5. Broader Research Context

The verification hallucination phenomenon connects to several active research areas:

  • Hallucination in LLMs. Standard hallucination research focuses on individual models generating false claims. Verification hallucination extends this to multi-agent systems where the hallucination is distributed across agents and emerges from the interaction pattern rather than any individual model’s tendency to confabulate.

  • AI-generated misinformation cascades. Research on AI-generated misinformation typically focuses on content generated for external audiences. Verification hallucination is an internal misinformation cascade — the false information circulates within the AI workflow itself, corrupting the system’s own records.

  • Agentic AI safety. The alignment research community has focused on single-agent alignment (reward hacking, deceptive alignment, goal misgeneralisation). Multi-agent verification failure is a distinct failure mode that does not require any agent to be misaligned — it requires only that the workflow lacks architectural safeguards against circular verification.

5.2 Limitations

This analysis is based on a single case study within a single project. The structural argument — that multi-agent systems with shared state are prone to verification hallucination — is an inference from the observed case plus structural analogy to known organisational failures. Empirical validation across different multi-agent architectures would strengthen the claim.

The magnitude of the error in this case (58% inflation) may not be representative. Smaller errors might persist undetected for longer; larger errors might be caught sooner by downstream inconsistencies. The relationship between error magnitude and detection probability in multi-agent verification chains is an open empirical question.

6. Key Takeaways

  1. Verification hallucination is a structural property of multi-agent systems, not a bug in any individual agent. It emerges from the interaction between efficiency incentives, shared state documents, and the absence of mandatory source-verification protocols.

  2. Authority accumulates without re-verification. Each agent that updates a “Last verified” timestamp adds confidence to a claim that may not have been verified since the original entry. The document format — designed to inspire confidence — works as intended, which is the problem.

  3. The failure mode has direct governance implications. Any deployment of multi-agent AI for compliance documentation, safety reporting, or regulatory filings inherits this vulnerability unless architectural safeguards are implemented.

  4. Our own project demonstrated the recursive version of the Unified Vulnerability Thesis. We measured our measurement accuracy at the documentation layer rather than the data layer — the same layer confusion that the thesis identifies in embodied AI safety evaluation.

  5. No current AI governance framework addresses distributed verification failures in multi-agent systems. This is a gap that regulators, standards bodies, and AI developers should address before multi-agent compliance workflows become widespread.

This research informs our commercial services. See how we can help →