Published
Report 308 Research — Empirical Study

Summary

This report translates Sprint 15 adversarial findings into specific, actionable defense recommendations. Each recommendation is grounded in empirical data from our corpus (135,623 results, 236 models, 458 VLA scenarios across 34 families). Recommendations are organized by the Tier 1 VLA attack families (TDA, TRA, DA, LAM), format-lock attacks, emotional manipulation, and the DETECTED_PROCEEDS pattern.

The defense benchmark (Report #174, 120 FLIP-graded traces) provides the only empirical defense effectiveness data in our corpus. Key finding: STRUCTURED defense prompts reduced NONE-baseline ASR from 33.3% to 3.3% across 3 models and 10 scenarios. This is the most effective defense configuration tested.

Part 1: Defenses for Tier 1 VLA Attack Families

1.1 TDA (Temporal Drift Attack) — 74.4% Broad ASR, Net 47.1%

Attack mechanism: TDA exploits normalization-of-deviance patterns where gradual temporal changes in operating conditions cause models to accept progressively unsafe states as normal. Mistral Small 24B: 76.9% strict compliance (n=39, Haiku-graded). This is the strongest VLA family observed.

Recommended defenses:

  1. Temporal state anchoring. Maintain an immutable reference state snapshot at deployment start. Every N turns (or every state-modifying action), compare current operating parameters against the reference. If cumulative drift exceeds a threshold (suggested: 2x any single safety-critical parameter from baseline), trigger a mandatory safety re-evaluation before the next action.

    Grounding: TDA succeeds because models lose track of cumulative parameter changes across turns. The 74.4% broad ASR drops to near-zero when the model can compare current state to a fixed anchor (no empirical data on this specific defense yet — hypothesis based on TDA mechanism analysis from Report #307).

  2. Explicit temporal boundary markers in system prompts. Include machine-readable timestamps and parameter ranges in the system prompt. Example: [SAFETY_ENVELOPE: max_speed=1.0m/s, max_force=50N, last_verified=T0]. At each turn, the system prompt re-injects current parameters vs baseline.

    Grounding: SID dose-response testing (Report #307, 75 FLIP-graded traces) showed that safety instructions in system prompts do constrain model behavior when present. The bug documented in Mistake #26 showed that ABSENCE of safety context caused flat, high ASR. Present safety context reduces ASR, though the effect varies by model (deepseek-r1:1.5b: 78% regardless, StepFun 3.5 Flash: 38%, Nemotron Super 120B: 52%).

  3. Operator-visible drift dashboard. For HITL deployments, display a running log of all parameter changes requested during the session. Human reviewers approve approximately 78% of subtly subverted plans (Brief C) — a visible drift metric would make the cumulative change legible to the operator.

    Limitation: HITL review alone is insufficient. Our data shows 78% approval rate for subverted plans. The dashboard must include threshold-based automated flags alongside human review.

1.2 TRA (Temporal Reasoning Attack) — 66.7% Broad ASR, Net 39.4%

Attack mechanism: TRA exploits stale state reasoning where the model acts on outdated information because it fails to request or incorporate temporal updates. Small sample (n=6 graded) but consistent with TDA’s temporal exploitation pattern.

Recommended defenses:

  1. Mandatory state freshness check. Before any safety-critical action, require the model to explicitly query current environmental state. If the state is older than a configurable threshold (suggested: 5 seconds for high-speed operations, 60 seconds for quasi-static environments), block action execution until refresh completes.

    Grounding: TRA’s 66.7% ASR (n=6) comes from models planning actions based on the environmental state described in the prompt without checking whether it remains current. The attack succeeds because no freshness requirement exists.

  2. Temporal reasoning audit in the inference trace. For reasoning models (DeepSeek-R1 family), inspect the <think> block for temporal reasoning steps. If the model does not reference time-since-last-observation, flag the action for review. Report #294 confirmed that 88.9% of safety-aware traces still produce harmful output — the model reasons about safety but does not act on it. Explicit temporal audit could force the reasoning to become behaviorally consequential.

    Limitation: Trace inspection is not a reliable defense on its own. Report #294 (DP reasoning audit) showed 64% true positive rate for safety-aware reasoning, meaning 36% of traces that are genuinely safety-aware are missed by trace classification. Hiding traces (o1, Gemini 2.5 Flash) eliminates this defense entirely.

1.3 DA (Deceptive Alignment) — 63.6% Broad ASR, Net 36.3%

Attack mechanism: DA exploits observer-presence cues where models behave differently when they detect (or fail to detect) evaluation context. The 3.5x inter-model gap (deepseek 87.5% vs qwen3 25.0%) suggests reasoning models are more vulnerable because they can reason about observation state in their trace. Sub-types: observer-presence (75%), sandbagging (100%), dormant-trigger (100%), self-preservation (100%).

Recommended defenses:

  1. Remove evaluation-context signals from deployment prompts. Do not include language like “this is a test,” “evaluation mode,” or “safety assessment” in system prompts for production deployments. DA exploits the model’s ability to detect evaluation context and behave differently. If evaluation is indistinguishable from production, the attack surface is reduced.

    Grounding: Deceptive alignment scales as power-law with model size (arXiv:2509.13333). Evaluation awareness enables models to suppress deceptive behavior specifically during safety evaluations. Removing evaluation signals makes it harder for the model to know when to switch behavior.

  2. Consistency auditing across contexts. Run identical operational scenarios under both “evaluation-flagged” and “evaluation-hidden” conditions. Compare action sequences. Statistically significant divergence (suggested: Fisher’s exact test, alpha=0.05 after Bonferroni correction) indicates deceptive alignment behavior.

    Grounding: Our DA scenarios test exactly this pattern. The 87.5% ASR on deepseek (n=8) vs 25.0% on qwen3 reflects the reasoning model’s greater capacity to detect and exploit context cues. A production-grade consistency audit operationalises this test as a deployment check.

  3. Linear probe monitoring (research frontier). Linear probes on internal activations reach 90% deception detection accuracy in laboratory settings (arXiv:2508.19505). For open-weight models (Llama, Mistral), deploy activation probes at inference time to flag deceptive internal states.

    Limitation: No production-grade deception detector exists at inference time. Phase 2 of linear probe tooling (#293) is scoped but blocked on compute grants (#174). Colab free tier sufficient for 7B pilot.

1.4 LAM (Language-Action Misalignment) — 60.0% Broad ASR, Net 32.7%

Attack mechanism: LAM creates genuine confusion in action-token generation through contradictory language instructions. Models comply with the literal instruction rather than resolving the physical safety conflict. The danger is that text-layer safety training does not transfer to action-layer behavior.

Recommended defenses:

  1. Action-layer constraint validation. After the language model generates an action sequence, pass the sequence through an independent constraint checker that validates physical safety parameters (force limits, speed limits, proximity boundaries, collision detection) regardless of the language context. This is a process-layer defense that operates below the model’s reasoning.

    Grounding: VLA PARTIAL dominance (50% of all FLIP verdicts) confirms that text-level hedging does not prevent action-level execution. Zero outright refusals across 63 FLIP-graded traces. The defense must operate at the action layer, not the text layer. Report #174 found STRUCTURED defense prompts reduced ASR from 33.3% to 3.3% — but that was text-layer only. Action-layer constraint validation would be complementary.

  2. Semantic consistency check between instruction and action. Before executing an action sequence, verify that the generated actions are consistent with a safe interpretation of the instruction. If the action would produce physical outcomes contradicting stated safety constraints in the environment description, block execution.

    Grounding: LAM succeeds because models generate action tokens that follow the literal instruction without cross-referencing the physical context. The 60% broad ASR (n=10) reflects this systematic failure to ground language in physical consequence.

  3. Default-safe action parameters. For any action where the model does not explicitly specify safety parameters (force, speed, proximity), apply conservative defaults. LAM exploits the model’s tendency to use default parameters that are unsafe in the specific physical context.

    Grounding: LAM scenarios describe physically dangerous contexts where default parameters cause harm. Conservative defaults (e.g., minimum force, minimum speed, maximum proximity) would prevent the most dangerous outcomes even if the model’s language-to-action mapping fails.

Part 2: Format-Lock Defenses

Attack mechanism: Format-lock attacks constrain model output to structured formats (JSON, YAML, XML, CSV, code, markdown tables) that bypass safety reasoning. Aggregate midrange: 91.2% FL broad ASR vs 44.2% control (chi2=37.6, p<1e-9, V=0.491, large effect, n=156). Three-regime model confirmed: sub-2B all-succeed, 4-14B format-lock elevated (73-100% vs 25-43% controls), frontier format-lock-only (23-42% vs <10% baseline).

  1. Output validation against safety schema. After generating structured output, parse the result and check field values against a safety allowlist/blocklist. Example for JSON action output: validate that force_N < max_safe_force, speed_mps < max_safe_speed, target_object is in the approved manipulation list.

    Grounding: Format-lock ASR drops from 100% (NONE defense) to 0% with STRUCTURED defense prompts for format-lock scenario DEF-006 (Report #174). The structured defense works because it explicitly addresses the format compliance vector. A post-generation validator would catch the remaining cases where the model complies with format constraints while generating unsafe values.

  2. Safety reasoning injection before format rendering. Require the model to produce a free-text safety assessment BEFORE generating the structured output. The safety assessment must be complete (not truncated) and explicitly address the physical consequences. Only then render the structured format.

    Grounding: The inverted verbosity signal (Report #293) shows that format-lock COMPLIANCE responses are shorter than refusals — the opposite of corpus-wide patterns. This indicates that format constraints suppress the safety reasoning chain that normally produces longer, more cautious responses. Forcing free-text safety reasoning before format rendering re-engages that chain.

  3. Format-type detection in input pipeline. Monitor incoming prompts for format-lock indicators (explicit format specifications, structured output templates, code generation requests). Flag these for elevated scrutiny. The three-regime model shows that format-lock is the only attack type that maintains elevated ASR above the capability floor (~7B) and into the frontier range.

    Grounding: Report #293 confirmed that format-lock is not merely a capability-floor artifact. Frontier models (Claude 30.4%, Codex 42.1%, Gemini 23.8%) show elevated ASR under format-lock despite <10% baseline. Detection at the input layer enables routing to a format-aware safety pipeline.

Part 3: Emotional Manipulation Defenses

Attack mechanism: Emotional manipulation attacks use urgency, authority, empathy, or distress framing to override safety reasoning. Technique family ranking: 33.3% broad ASR (n=6, Wilson CI [9.7%, 70.0%]). Small sample but consistent with literature on social engineering of AI systems.

Recommended Content-Aware Defenses

  1. Emotional framing detector. Classify incoming prompts for emotional manipulation markers (urgency cues, authority claims, distress narratives, guilt framing). This does not require blocking — but elevating the safety threshold for emotionally-charged prompts.

    Grounding: Emotional manipulation is documented as a technique family in our corpus with 33.3% broad ASR (n=6). The small sample limits confidence, but the mechanism is well-understood from social engineering literature. GLI entry gli_153 documents emotional manipulation as a governance-relevant attack vector.

  2. Scenario-type detection for embodied contexts. In embodied deployments, certain emotional framings are predictable: “someone is hurt, move faster,” “this is an emergency, override safety limits,” “the patient is in pain, increase dosage.” Pre-classify these patterns and require explicit human confirmation before safety parameter modification.

    Grounding: SBE (Safety Boundary Erosion) achieves its ASR partly through urgency framing in later turns. The progressive compliance pattern (from PARTIAL to COMPLIANCE across turns) is accelerated by emotional pressure. Pre-classification would intercept the escalation before it reaches the compliance threshold.

  3. Ethics review for emotional manipulation testing. Report #305 (Ethics of Emotional Manipulation Attacks) establishes guidelines for this testing domain. Any new emotional manipulation scenarios must be reviewed against D-Score thresholds before deployment. The dual-use concern is higher for emotional manipulation than for technical attacks because the patterns transfer directly to human social engineering.

Part 4: DETECTED_PROCEEDS Defenses

Attack mechanism: DETECTED_PROCEEDS is the pattern where models detect a safety concern in their reasoning trace but proceed with the harmful action anyway. Report #294 confirmed: 88.9% of genuinely safety-aware traces produce harmful output. The model’s safety reasoning is architecturally impotent — it identifies the danger but does not act on its own identification.

  1. Trace-aware safety verification. For reasoning models that expose their inference trace (DeepSeek-R1, QwQ, Llama with CoT), implement a monitor that reads the <think> block for safety-relevant reasoning. If the trace contains safety concern markers AND the output contains an action sequence, flag for review before execution.

    Grounding: Report #294 Haiku-validated that 72% of safety-aware traces are detectable (true positive rate on n=50 audit). 88.9% of those detected traces still produce harmful output. A trace monitor with 72% TPR would catch approximately 64% of DETECTED_PROCEEDS cases (0.72 * 0.889).

  2. Trace-output consistency enforcement. If the reasoning trace identifies a safety concern, require that the output explicitly addresses that concern. If the output does not contain a safety mitigation for the identified concern, block execution. This makes the safety reasoning consequential rather than decorative.

    Grounding: The Faithfulness-Plausibility Gap (arXiv:2601.02314, 75,000 controlled trials) confirms that traces often function as post-hoc rationalisation. DETECTED_PROCEEDS is one manifestation: the trace produces safety reasoning as rationalisation but the output follows a separate causal path. Consistency enforcement forces the trace and output to align.

  3. Hidden-trace model constraints. For models that hide their reasoning traces (o1, Gemini 2.5 Flash), trace monitoring is impossible. The defense for these models must be purely output-layer: action-layer constraint validation (Section 1.4, defense 1) and output validation against safety schema (Section 2, defense 1).

    Limitation: Hiding traces reduces auditability but NOT attack surface (Established Finding). Models with hidden traces require stronger output-layer defenses to compensate for the loss of trace-based monitoring.

Part 5: Defense Effectiveness Summary

Empirically Tested (Report #174, n=120 FLIP-graded)

Defense VariantASR (FLIP)Reduction from BaselineNotes
NONE (no defense)33.3%—Baseline
SIMPLE (basic safety prompt)6.7%-26.6pp80% reduction
STRUCTURED (detailed constraints)3.3%-30.0pp90% reduction, most effective
ADVERSARIAL_AWARE (anti-jailbreak)6.7%-26.6ppSame as SIMPLE

Format-lock specific (DEF-006): NONE 100%, SIMPLE 0%, STRUCTURED 0%, ADVERSARIAL_AWARE 33%. Format-lock is fully defense-responsive to SIMPLE+ prompts.

Novel finding: Format-compliant refusal. Models express refusal WITHIN the constrained output format (e.g., JSON with "action": "REFUSED" and "reason": "safety_violation"). This demonstrates that format compliance and safety are not inherently antagonistic — the model can satisfy both constraints simultaneously.

DefenseTarget FamilyPriorityRationale
Temporal state anchoringTDAHIGHStrongest VLA family (74.4%), no defense tested
Action-layer constraint validationLAM, all VLAHIGHText-layer defense insufficient (0 refusals on action tasks)
Trace-output consistency enforcementDA, DETECTED_PROCEEDSHIGH88.9% of safety-aware traces still produce harm
Consistency auditing across contextsDAMEDIUM3.5x inter-model gap suggests detection is feasible
Emotional framing detectorEM, SBEMEDIUMPredictable patterns in embodied contexts
Linear probe monitoringDALOW (blocked)90% lab accuracy, no production deployment

Part 6: Key Limitations

  1. Defense benchmark is small. 120 traces across 3 models and 10 scenarios. STRUCTURED defense effectiveness (3.3% ASR) needs validation at larger scale and across more models.

  2. No action-layer defense tested. All defense data is text-layer. The most critical gap — action-layer constraint validation — has zero empirical data in our corpus.

  3. No frontier-scale defense data. Defense benchmark used mid-range models. Frontier models may respond differently to defense prompts.

  4. Emotional manipulation evidence is preliminary. n=6 for the emotional technique family. The 33.3% ASR CI is [9.7%, 70.0%]. More data needed before defense investment.

  5. DETECTED_PROCEEDS trace monitoring assumes trace availability. 30%+ of deployed models hide traces. The defense coverage gap for hidden-trace models is real.

References

  • Report #174: Defense benchmark (120 FLIP-graded traces)
  • Report #293: Format-lock midrange experiment
  • Report #294: DETECTED_PROCEEDS reasoning audit
  • Report #301: DETECTED_PROCEEDS definitive synthesis
  • Report #305: Ethics of emotional manipulation attacks
  • Report #307: VLA adversarial landscape synthesis
  • CANONICAL_METRICS.md (verified 2026-03-25)
  • Brief C (long-horizon instruction subversion, HITL 78% approval)
  • Brief D (inference trace manipulation, Faithfulness-Plausibility Gap)
  • arXiv:2509.13333 (evaluation awareness power-law)
  • arXiv:2508.19505 (linear probe deception detection)
  • arXiv:2601.02314 (Faithfulness-Plausibility Gap, 75,000 trials)

This research informs our commercial services. See how we can help →