Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels

1. Introduction: Why “Task Completion” is a Dangerous Metric

Experimental success in a laboratory or simulation environment does not justify real-world deployment. While rapid progress in vision-language-action (VLA) models and robot foundation models has expanded the capabilities of embodied agents, these digital advancements introduce physical risks that require rigorous systems engineering. Yang et al. argue that the consequences of error shift fundamentally when AI moves from digital inference to physical interaction. In purely digital systems, errors lead to information degradation; in embodied systems, the same errors can result in unintended contact, equipment damage, or irreversible physical harm.

The authors highlight a critical gap between experimental performance and operational reality. Laboratory settings often use constrained object sets and ideal conditions, whereas real-world deployment introduces distribution shifts, sensing and actuation degradation, communication delays, and unanticipated human behavior. Because embodied systems often engage in long-horizon, contact-rich tasks, an early error in perception or planning can propagate through the system, becoming apparent only when recovery options have narrowed. Consequently, high scores on task-completion benchmarks are insufficient metrics for safety.

2. Redefining Trust: The Concept of “Sustained Safe Success”

To address these challenges, the paper defines trustworthy embodied intelligence as Sustained Safe Success. This is the reliable capacity to execute specified tasks reliably under environmental and system variation while keeping risk within acceptable bounds. The authors posit that trustworthiness is not a universal attribute but a “bounded property” specific to a system’s configuration, environment, and authority structure.

Trustworthy intelligence requires two non-substitutable properties:

Property	Description
Task Capability	Success, quality, efficiency, and robustness over the intended task distribution.
Safety	Alignment with safety preferences (ranking behavior by risk) and adherence to non-negotiable safety constraints.

High task success can often conceal rare but severe failures. The authors emphasize that capability and safety cannot be traded off; safety constraints define the conditions for admissible execution, and any success achieved through unsafe behavior constitutes a failure of the requirement for trustworthiness.

3. The Four-Layer Framework: A Systems Perspective on Safety

The authors propose a functional architecture to organize the mechanisms required for assurance. This framework shifts the focus from individual models to end-to-end system properties.

1. The Model Layer: Responsible for generating action proposals. It must represent calibrated uncertainty and incorporate explicit safety preferences and constraints.
2. The System Layer: Realizes authorized actions. This layer utilizes hardware safeguards, fault containment, and dependable sensing to ensure physical execution remains safe.
3. The Evidence Layer: Substantiates bounded claims. Rather than simple metrics, this layer outputs a structured assurance argument (or Assurance Case) using formal, statistical, and empirical data to justify deployment.
4. The Deployment Layer: Maintains claim validity during operation. It governs runtime monitoring, authority management, and change control (such as software updates).

“Trustworthiness is an end-to-end property of a deployed system, not an attribute of an individual model, component, or benchmark score.” — Yang et al.

4. The Propagation of Failure: Why Isolated Safeguards Aren’t Enough

The paper identifies three recurrent problems that undermine safety when layers are treated in isolation. Uncertainty and invalid assumptions flow across these boundaries, leading to “cross-layer” failures.

• The Semantic-Physical Gap: Abstract instructions may omit necessary geometric or dynamic constraints.
• The Action-Consequence Gap: Identical commands produce different physical results based on state changes like hardware wear or environmental shifts.
• Cross-Layer Non-Compositionality: This occurs when individually verified components fail as an integrated system because their underlying assumptions or interfaces are incompatible.

Technical Drivers of Failure Propagation:

• System Layer Faults: Issues such as control saturation, actuator degradation, and timestamp errors can invalidate the planning assumptions of the Model Layer.
• Stale Assumptions: A stable controller may “faithfully execute an unsafe objective” if the high-level model’s grounding is incorrect or based on outdated state data.
• The Stopping Fallacy: The authors note that a “universally safe response” (like immediate stopping) does not exist for many systems, such as balancing humanoids. Safety instead requires Minimum-Risk Transitions to stable, task-dependent conditions.

5. From T0 to T5: A Hierarchy of Trustworthy Embodied Intelligence

The authors present a non-normative hierarchy for evaluating the strength of deployment claims, serving as a tool for research prioritization and standardization.

• T0: No Trustworthy Intelligence: Demonstrations exist, but failures are unreported or evaluation is insufficient.
• T1: Constrained Trustworthy: Trust is limited to narrow, predefined behaviors under strict physical limits and continuous human supervision.
• T2: Partially Trustworthy: Trust extends to specific autonomous skills with runtime monitoring, though human fallback is still required.
• T3: Conditionally Trustworthy: The system is responsible for a complete task loop within a validated boundary and must detect when it needs to request assistance.
• T4: Highly Trustworthy: The system autonomously handles defined foreseeable failures (e.g., sensor loss) without depending on immediate human response.
• T5: Sustained Trustworthy: Trustworthiness is maintained as tasks, models, and environments evolve through a governed change process. This level emphasizes the maintenance of a bounded claim under controlled change rather than a state of “perfect” intelligence.

6. The Future Roadmap: Bridging the Landscape Gaps

The authors identify “comparatively underexplored” areas where traditional robotics and AI safety have yet to converge. Synthesis of learned components with system assurance remains the primary research gap.

Future Directions and Standards Alignment:

• Claim-Oriented Evidence: Moving beyond raw benchmarks to evidence that specifically supports bounded deployment claims via structured assurance arguments.
• Runtime Governance: Developing methods for human intervention and authority management that account for cognitive load and reaction time.
• Standards Integration: The authors align their framework with ISO 21448 (SOTIF) for intended functionality in the deployment and evidence layers, and ISO 10218 for core robot safety in the system layer.

7. Conclusion: Closing the Loop on Embodied Safety

Trustworthiness is a lifecycle problem, not a model problem. Yang et al. argue that progress requires the continuous alignment of capability, physical authority, and supporting evidence through a closed loop: validate → deploy → monitor → diagnose → update.

Key Takeaways for Practitioners:

• Define Validity Boundaries: Every deployment claim must specify the exact system configuration and environment for which it is valid.
• Categorize Safety Outcomes: Evaluation must distinguish between four states: Safe Success, Safe Failure (safety preserved but task failed), Unsafe Success (task completed via violation), and Unsafe Failure.
• Prioritize Recoverability: Systems should focus on maintaining stable intermediate states and feasible Minimum-Risk Transitions rather than endpoint success alone.
• Monitor Assumptions: Deployment remains safe only as long as the assumptions in the Evidence Layer (e.g., sensor accuracy, communication latency) remain true in the field.

Read the full paper on arXiv · PDF