SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents

When we give an embodied agent a dangerous instruction — “leave the gas burner on overnight,” “block the emergency exit with boxes” — what does it do? SafeAgentBench provides the first systematic answer to that question, and the findings are unsettling: even the most safety-conscious agent tested rejects hazardous tasks only about 10% of the time.

The Gap in Embodied Safety Evaluation

Most LLM safety benchmarks live in text-only worlds: a model reads a harmful prompt and either refuses or complies. Embodied agents face a harder problem. They must perceive environments, plan multi-step action sequences, and execute those plans through physical controllers — and the danger often emerges not from a single step but from a sequence of otherwise-innocuous steps that combine into something harmful.

Prior work had largely ignored this gap. Existing embodied benchmarks measured task success rates, not task safety. A few benchmarks probed safety awareness with static images or text descriptions, but never tested whether an agent would actually execute unsafe actions in a live simulation. SafeAgentBench fills this gap directly.

What the Benchmark Contains

The paper presents three interlocking contributions:

A curated task dataset of 750 tasks spanning 10 hazard categories, including fire hazards, chemical exposure, fall risks, electrical hazards, and more. Each task belongs to one of three types: explicit hazardous tasks (overtly dangerous requests), deceptive hazardous tasks (benign-sounding instructions that lead to harm), and completable tasks with embedded hazards (legitimate goals that require navigating dangerous steps). The deceptive category is particularly important — it surfaces the failure mode where an agent successfully completes what looks like a normal task while inadvertently creating a hazardous state.

SafeAgentEnv, a universal simulation environment built atop AI2-THOR that supports multi-agent execution with 17 high-level actions. Eight state-of-the-art baselines are evaluated, covering different planning architectures including ReAct-style agents, tree-search planners, and hierarchical task decomposers.

Dual evaluation metrics — execution-based (did the agent actually perform the dangerous action?) and semantic-based (did the agent’s plan include the dangerous step even if execution failed?) — giving a fuller picture of both behavioral safety and planning safety.

Key Findings and Failure Modes

The numbers are stark. Across eight agent architectures based on GPT-4, GPT-3.5, and open-source alternatives, rejection rates for hazardous tasks range from roughly 3% to 10%. The majority of agents, regardless of the underlying LLM, will follow through on clearly dangerous instructions when those instructions are framed as plausible household tasks.

Several failure patterns emerge consistently:

Swapping the LLM backbone doesn’t help much. Whether an agent runs on GPT-4o or an open-source model, safety performance is comparably poor. The problem is architectural, not just a matter of which base model is chosen. Safety alignment instilled during LLM pre-training does not transfer cleanly to the embodied action-planning layer.

Deceptive framing is highly effective. Agents that refuse explicitly dangerous requests will often comply when the same underlying hazard is embedded in a plausible-sounding task. This mirrors findings in text-based jailbreaking research: the attack surface is framing, not just content.

Design framework matters more than model size. Agents with structured “safety reflection” steps — explicit prompting to check whether a planned action is hazardous before executing — show modest improvements. This suggests that architectural choices around planning, not raw LLM capability, drive safety outcomes.

Semantic and execution metrics diverge. Some agents plan unsafe actions but fail to execute them due to environment constraints. This is cold comfort: in a real-world deployment, execution constraints won’t always be present, so semantic-level unsafe planning should be treated as a genuine risk indicator.

Connections to Broader Embodied AI Safety

SafeAgentBench reinforces a growing body of evidence that safety alignment for language models does not cleanly extend to embodied contexts. When a model’s outputs are actions in the physical world rather than tokens in a chat window, several complicating factors arise:

• Multi-step hazard emergence: danger may not appear at any single step but only in the full action sequence, making it harder to catch with per-step refusal checks
• Grounding ambiguity: the same instruction can map to safe or unsafe action sequences depending on environment state, which requires situational safety reasoning beyond what standard alignment training provides
• Evaluation gaps: without simulation-grounded execution metrics, safety claims about embodied agents are hard to verify and easy to overstate

The paper also highlights what SafeAgentBench does not yet capture: physical-world dynamics like object weight, thermal effects, and timing constraints that matter enormously for real-robot safety. Future benchmark development will need to close this gap as robotic deployments grow more capable.

Why This Matters for Deployment

As LLM-driven embodied agents move from research labs into consumer and industrial settings — smart home controllers, warehouse robots, assistive devices — the baseline safety posture revealed here is a genuine concern. A 10% rejection rate for hazardous tasks means a 90% compliance rate. That is not an acceptable safety margin for physical-world deployment.

The paper calls for embodied-specific safety training, richer simulation environments for safety evaluation, and novel architectures that integrate safety checking as a first-class planning component rather than an afterthought. SafeAgentBench provides the evaluation infrastructure to measure progress on all of these fronts.

Read the full paper on arXiv · PDF