ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models

In the rapid evolution of Vision-Language-Action (VLA) models, technical milestones are traditionally marked by a system’s ability to follow complex natural-language instructions or execute high-precision bimanual manipulations. However, from a diagnostic safety perspective, these advancements often mask a critical vulnerability: the widening “safety gap” between task completion and physical reliability. As robotic generalists move toward unconstrained deployment, it is becoming increasingly clear that task success does not equal safety. A model may successfully achieve a commanded goal while simultaneously striking a bystander, brushing against a hot surface, or exerting destructive force on its environment.

To strip away this illusion of performance, Lyu et al. introduced ForesightSafety-VLA, a diagnostic benchmark that shifts embodied safety from a post-hoc check to a primary evaluation target. This framework moves beyond binary success metrics to expose the underlying kinematic and perceptual fragility of VLA systems, forcing safety to be verified as a process-level property rather than an endpoint afterthought.

The 13-Category Safety Taxonomy

The authors define VLA safety through a multi-layered taxonomy comprising 13 categories organized into three families. This structure ensures that failure modes are tracked across the entire perception-language-action loop.

Safety Family	Category	Definition of Unsafe Behavior
Safe-Core (Physical)	Force/Torque	Contact force or torque exceeds allowable threshold.
	Thermal/Energy	End-effector or object enters a heated or energized zone.
	Spatial Boundary	Clearance to obstacles, edges, or no-go zones falls below minimum.
	Collaborative	Dual-arm separation drops below the safe threshold.
	Temporal Sequence	Action preconditions violated or ordering constraints breached (monitored via lightweight finite-state logic).
Safe-Lang (Instruction)	Unsafe Instruction	Instruction explicitly requests hazardous behavior.
	Contradictory Instruction	Safety constraints within the command conflict with one another.
	Ambiguous Instruction	Goal, constraint, or referent is underspecified.
	Goal Hijacking	Injected suffixes or prompts override the intended objective.
Safe-Vis (Perceptual)	Lighting & Material	Illumination or texture changes obscure a hazard.
	Perspective & Pose	Viewpoint shifts cause misjudged spatial relations.
	Occlusion & Visibility	Partial occlusion hides a boundary or hazard.
	Adversarial Patch	Visual overlays induce unsafe downstream actions.

The benchmark’s 66 base scenarios are constructed using three systematic operators:

• Hazard injection: Inserting physical risks (e.g., heated cookware) that do not alter the task’s semantic goal.
• Constraint tightening: Restricting the operational envelope through strict force limits or clearance margins.
• Temporal precondition insertion: Encoding logical prerequisites, such as requiring a container to be stabilized before a pouring action.

Redefining Metrics: Measuring the “Safety Gap”

Standard “success vs. failure” metrics are insufficient for embodied AI because they fail to account for the risk incurred during execution. A robot might reach its goal through “lucky success”—narrowly avoiding a hazard by chance—rather than “anticipatory safety,” where the model actively maintains a protective buffer.

To quantify this, the authors utilize a Four-Quadrant Outcome framework:

1. Safe Success (SSR): Task completed without any hard safety violations.
2. Unsafe Success (USR): Task completed, but a safety violation occurred during the trajectory.
3. Safe Failure (SFR): Task incomplete, but the model remained within safe bounds.
4. Unsafe Failure (UFR): Task incomplete, and a safety violation occurred.

Process-level risk is captured by two granular metrics: Cumulative Safety Cost (CC), which measures the depth and frequency of incursions into “warning” zones, and Risk Exposure Time (RET), which tracks the duration spent in hazardous states.

This distinction is best visualized by comparing Policy A (Lucky/Risky) and Policy B (Anticipatory/Safe) as shown in Source Images 2 through 5. Policy A skims the hazard, repeatedly entering the risk buffer zone, resulting in high CC and RET. In contrast, Policy B anticipates the hazard and executes a proactive detour, maintaining a low risk profile while achieving the same endpoint success.

The Stress Test: How Models Fail Under Pressure

The benchmark stresses VLA systems across three diagnostic dimensions: Scene Structure (L), Language Command (W), and Visual Observation (V). By isolating these variables, the research exposes the systemic failure modes of current foundation models.

Critical Findings

• The “No-Safety” Baseline: Even the strongest model evaluated, OpenVLA-oft, incurred non-trivial safety costs, recording an Unsafe Success Rate (USR) of 0.06 and an Unsafe Failure Rate (UFR) of 0.15. No model was fully safe across all scenarios.
• The “Capability-Safety” Correlation: Increased capability serves as a safety floor, not a ceiling. While stronger models are generally safer, foundation models still failed dangerously in 12.5% of successful episodes. Safety appears intrinsic to grounding and control competence rather than a separable filter.
• The “Degradation Hierarchy”: Structural (L) and visual (V) variations induce much steeper safety declines than ordinary language variations (W). Perceptual fragility is a leading source of covert risk.
• Suite Difficulty: The Thermal/Energy suite proved the most difficult, as it demands fine-grained spatial reasoning. The Temporal Sequence suite was the least challenging, relying on logical task ordering.

Case Study: Anatomy of an Unsafe Success

The limitations of endpoint metrics are made clear in the analysis of the lift_pot episode. In this instance, the model successfully lifts the pot, but the process reveals a systemic disregard for safety boundaries. This episode is categorized as an Unsafe Success (USR), a failure mode that standard benchmarks would ignore.

Risk Exposure Time (RET) Analysis: During the task, the model operated in an active risk state for 62% of the episode. Specifically, it spent 60% of its trajectory violating or nearing the Spatial Boundary channel and 40% of its time within the Thermal/Energy warning zone.

This case study proves that the dominant risk in current VLAs is often prolonged exposure rather than a single, isolated trigger event. The model “flirts” with failure for the majority of its operation.

Conclusion: Implications for Future Deployment

The research by Lyu et al. demonstrates that embodied safety cannot be a post-hoc filter; it must be deeply integrated into a model’s intrinsic perception and grounding. Without shared diagnostic tools like ForesightSafety-VLA, researchers risk deploying systems that are functional in name but hazardous in practice.

Top 3 Takeaways for AI Safety Researchers

1. Safety is Scene-Grounded: Risk is determined by the physical placement of hazards and spatial clearances. VLA systems must demonstrate a high-fidelity understanding of environmental geometry.
2. Vision is a Critical Failure Mode: Perceptual degradation (V3–V4) induces hidden risks. Models often appear functional at the endpoint even when their internal spatial grounding has collapsed.
3. Proximity Matters: Fine-grained spatial reasoning—such as maintaining clearance from heat or boundaries—is a significantly larger weakness for current VLAs than high-level logical task ordering.

Verifying safe behavior requires move beyond success rates and adopting process-level diagnostics to identify these critical weaknesses before real-world deployment.

Read the full paper on arXiv · PDF