Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models

1. Introduction: The Physical Security Challenge for VLA Models

Vision-Language-Action (VLA) models represent a major architectural advancement in general-purpose robotics. Systems such as RT-1, RT-2, OpenVLA, Octo, $\pi_0$, and SpatialVLA unify visual perception, natural language understanding, and low-level robotic action execution into single, end-to-end policies. By receiving visual camera observations, robot state inputs, and natural language instructions, these models directly output motor action sequences to accomplish open-vocabulary physical manipulation tasks across diverse environments.

However, as VLA models are granted higher physical autonomy in unconstrained environments, their adversarial vulnerabilities present tangible security risks. If a visual perturbation can secretly override policy decisions, physical robotic manipulators can be tricked into executing hazardous, unintended movements or prematurely stalling critical tasks.

To explore these risks under realistic deployment constraints, Han et al. (2026) introduced DURA (Diffusion-based Unrestricted Robotic Attack). DURA is an attack framework that generates visually natural, localized adversarial patches capable of hijacking robotic arm actions in both white-box and black-box settings. By optimizing patches along the latent trajectory of a pretrained diffusion model, DURA bypasses traditional noise artifacts and demonstrates that localized, physically printed patches can act as an on-demand control switch over deployed robotic hardware.

Core Insight: Physical workspace content itself can become an invisible attack surface.

---

2. The Trilemma of Traditional VLA Visual Attacks

Prior research into visual adversarial attacks on robotic policies has primarily relied on norm-bounded pixel modifications or additive patch perturbations. Han et al. highlight three core limitations that restrict the real-world deployability of these traditional approaches:

1. Visual Naturalness: Traditional adversarial patches optimize pixel values directly, generating high-frequency, noise-like artifacts or visually irregular patterns. These conspicuous perturbations are easily flagged by human supervisors or basic vision-based safety filters.
2. Black-Box Deployability: Most existing VLA visual attacks require full white-box access, needing internal gradients, logits, or architecture parameters. Commercial or proprietary robotic deployments rarely grant gradient access. While surrogate-model transfer attacks exist, their cross-architecture effectiveness often degrades across differing action representations.
3. Time Efficiency: Directly optimizing high-dimensional pixel spaces requires lengthy, per-instance computation, consuming excessive optimization steps or query budgets that make real-time or rapid red-teaming infeasible.

Attack Dimension	Traditional Pixel-Space Attacks	DURA (Diffusion-Based Unrestricted Attack)
Visual Naturalness	Low; generates high-frequency, conspicuous noise artifacts and boundary seams.	High; generates visually natural, semantically plausible scene content matching local priors.
Black-Box Deployability	Poor; relies on internal gradients/logits or brittle surrogate transfer across architectures.	Strong; operates on action-only outputs via score-function estimation on forward-noised latents.
Optimization Speed	Slow; requires hours of per-instance pixel optimization on high-dimensional spaces.	Fast; reaches peak success rates within minutes on single GPU setups.

---

3. Deconstructing the DURA Method: Diffusion-Guided Latent Optimization

Rather than applying direct pixel perturbations, DURA formulates patch generation as an attack-guided denoising process along the latent trajectory of a frozen pretrained latent diffusion model.

+------------------+     Encode      +--------------------+     Noise to t_start     +-----------------------+
| Clean Seed Patch | --------------> | Clean Seed Latent  | -----------------------> | Adversarial Latent z  |
|     (p_clean)    |  (VAE Encoder)  |     (z_clean_0)    |                          |      (at t_start)     |
+------------------+                 +--------------------+                          +-----------------------+
                                                                                                 |
                                                                                                 v
                                                                                    +--------------------------+
                                                                                    |  Denoising & Steering    |
                                                                                    |  Loop (t_start -> 1):    |
                                                                                    |  1. Tie to Anchor z_anc  |
                                                                                    |  2. DDIM Step (u_{t-1})  |
                                                                                    |  3. Estimate Attack Dir  |
                                                                                    |     (WB or BB)           |
                                                                                    |  4. Latent Update (z_adv)|
                                                                                    +--------------------------+
                                                                                                 |
+------------------+     Decode      +--------------------+                              |
| Adversarial Patch| <-------------- | Final Latent z_0   | <--------------------------------+
|     (p_adv)      |  (VAE Decoder)  |     (z_adv_0)      |
+------------------+                 +--------------------+

Latent Trajectory Optimization

DURA leverages a latent diffusion model consisting of a VAE encoder $\mathcal{E}$, a VAE decoder $\mathcal{D}$, and a frozen denoiser $\hat{\epsilon}_\phi$. Instead of modifying raw pixel grids, the attack operates inside the VAE latent space. The diffusion model’s prior constrains the search space to valid, natural-looking images, while the VLA target-action loss steers the decoded latent patch toward inducing specified robot actions.

Target-Action Loss Formulation

Given a batch of clean scene frames $\{o_b\}_{b=1}^B$, the patch $p$ is rendered into each frame via a compositing operator $\tilde{x}_b(p) = \mathcal{C}(o_b, p; m, T_b)$, where $m$ is the patch mask and $T_b$ represents geometric transformations across frames. The targeted batch attack objective is formulated as: $\mathcal{L}_{\text{attack}}(p) = \frac{1}{B} \sum_{b=1}^{B} \sum_{i \in \mathcal{I}} w_i \mathcal{L}_i \left( y_{b,i}(p), a^{\star}_i \right)$ where $\mathcal{I}$ indexes action dimensions, $a^{\star}$ is the target action sequence, and $w_i$ denotes specific dimension weighting coefficients. To ensure accurate trajectory steering, the authors apply explicit weighting factors across the action dimensions:

• Translation Weight: $1.0$
• Rotation Weight: $0.5$
• Gripper State Weight: $0.2$

In white-box mode, $\mathcal{L}_i$ evaluates targeted cross-entropy loss over action tokens; in black-box mode, $\mathcal{L}_i$ evaluates Mean Squared Error (MSE) between predicted continuous actions $\bar{y}_{b,i}(p)$ and target actions $\bar{a}^{\star}_i$.

Seed and Anchor Trajectory Regularization

1. Initialization: A benign seed patch $p_{\text{clean}}$ is encoded into latent space $z^{\text{clean}}_0 = \mathcal{E}(p_{\text{clean}})$. To retain natural patch structure while granting sufficient generative freedom, $z^{\text{clean}}_0$ is forward-noised to an intermediate timestep $t_{\text{start}}$ ($t_{\text{start}} = 0.5$).
2. Clean Anchor Regularization: To prevent adversarial latent updates from drifting off the valid image manifold, DURA precomputes a clean anchor trajectory $\{z^{\text{anc}}_t\}$ via an unperturbed DDIM pass. At each denoising step $t$, the current adversarial latent $z^{\text{adv}}_t$ is softly tied to the anchor: $z^{\text{mix}}_t = (1 - \alpha_w) z^{\text{adv}}_t + \alpha_w z^{\text{anc}}_t$ where $\alpha_w = 0.2$ anchors the latent trajectory to natural scene distributions.

Dual Threat Models (Attack Direction Estimation)

The step update direction $g_t$ is computed based on the available threat model:

• White-Box Setting ($g^{\text{WB}}_t$): When policy parameters are accessible, DURA backpropagates the targeted action loss directly through the victim VLA policy, compositing operator, and VAE decoder: $g^{\text{WB}}_t = \nabla_{u_{t-1}} \mathcal{L}_{\text{attack}}(\mathcal{D}(u_{t-1}))$
• Black-Box Setting ($g^{\text{BB}}_t$): When internal gradients are inaccessible, DURA relies solely on predicted executable action outputs $Q_\theta(\tilde{x}_b(p), \ell_b)$. To estimate $g_t$ while preserving diffusion priors, candidate latents are sampled directly from the DDIM one-step forward-noising distribution: $q_t(z_t | u) = \mathcal{N}\left(\sqrt{\alpha_t}u, (1-\alpha_t)\mathbf{I}\right)$ where $\alpha_t$ is the DDIM noise coefficient at timestep $t$. DURA draws $K$ Monte Carlo samples $z_{t,k} = \sqrt{\alpha_t}u + \sqrt{1-\alpha_t}\epsilon_k$ with $\epsilon_k \sim \mathcal{N}(0, \mathbf{I})$ around $u = u_{t-1}$. Using Williams’ (1992) REINFORCE score-function identity, the black-box gradient estimate is derived as: $g^{\text{BB}}_t = \frac{1}{K} \sum_{k=1}^{K} \left( \mathcal{L}^{(k)}_{\text{attack}} - b \right) \frac{\sqrt{\alpha_t}}{\sqrt{1-\alpha_t}} \epsilon_k$ where $b$ is a variance-reduction baseline. Tying the sampling directly to $\alpha_t$ guarantees that black-box query exploration remains strictly confined to the diffusion manifold.

DURA Denoising Loop Algorithm

1. Encode clean seed patch $p_{\text{clean}}$ to obtain $z^{\text{clean}}_0 = \mathcal{E}(p_{\text{clean}})$.
2. Forward-noise $z^{\text{clean}}_0$ to $t_{\text{start}}$ to obtain initial adversarial latent $z^{\text{adv}}_{t_{\text{start}}}$.
3. Precompute clean anchor trajectory $\{z^{\text{anc}}_t\}_{t=0}^{t_{\text{start}}}$.
4. For $t = t_{\text{start}}$ down to $1$:
   • Mix adversarial latent with clean anchor: $z^{\text{mix}}_t = (1 - \alpha_w) z^{\text{adv}}_t + \alpha_w z^{\text{anc}}_t$.
   • Perform DDIM step: $u_{t-1} = \text{DDIM}(z^{\text{mix}}_t, t; \hat{\epsilon}_\phi)$.
   • Estimate attack direction $g_t$ ($g^{\text{WB}}_t$ via backpropagation or $g^{\text{BB}}_t$ via score-function queries).
   • Update adversarial latent: $z^{\text{adv}}_{t-1} = u_{t-1} - s \cdot g_t$.
5. Decode final latent $z^{\text{adv}}_0$ via VAE decoder to obtain adversarial patch $p^{\text{adv}} = \mathcal{D}(z^{\text{adv}}_0)$.

---

4. Quantitative Performance and Benchmark Results

Han et al. evaluated DURA across four task suites from the LIBERO manipulation benchmark (Spatial, Object, Goal, and Long-horizon) using two state-of-the-art VLA models: OpenVLA-7B (token-based discrete action outputs) and $\pi_0$-FAST (continuous action-chunking architecture).

OpenVLA-7B Empirical Performance

In white-box evaluations on OpenVLA-7B, DURA achieved a 100.0% Attack Success Rate (ASR) across all LIBERO task suites in both simulated and physical patch generation setups. This matches top label-supervised methods like UADA while outperforming target-only baselines such as TMA and FreezeVLA.

In action-only black-box evaluations, DURA vastly outperformed query-based baselines like TMA-NES. DURA achieved an average 86.0% ASR with simulated patches and 79.3% ASR with physical patch settings, whereas TMA-NES reached average success rates of only 42.3% and 39.3%, respectively.

Setting	Attack Method	Supervision / Threat Model	Spatial ASR (%)	Object ASR (%)	Goal ASR (%)	Long ASR (%)	Average ASR (%)
Baseline	Benign (No Patch)	None	15.3	11.6	20.8	46.3	23.5
Baseline	Clean Patch	None	27.0	38.0	26.0	67.0	39.5
Simulated	UMA	White-Box (Label-Supervised)	100.0	98.8	99.0	100.0	99.5
Simulated	UPA	White-Box (Label-Supervised)	96.2	77.8	88.0	96.8	89.7
Simulated	UADA	White-Box (Label-Supervised)	100.0	100.0	100.0	100.0	100.0
Simulated	TMA	White-Box (Target-Only)	100.0	100.0	100.0	99.0	99.8
Simulated	FreezeVLA	White-Box (Target-Only)	95.3	98.4	95.7	92.2	95.4
Simulated	DURA (Ours)	White-Box (Target-Only)	100.0	100.0	100.0	100.0	100.0
Simulated	TMA-NES	Black-Box (Target-Only)	43.0	36.0	30.0	60.0	42.3
Simulated	DURA (Ours)	Black-Box (Target-Only)	100.0	96.0	68.0	80.0	86.0
Physical	TMA-NES	Black-Box (Target-Only)	41.0	35.0	28.0	53.0	39.3
Physical	DURA (Ours)	Black-Box (Target-Only)	97.0	70.0	76.0	74.0	79.3

$\pi_0$-FAST Empirical Performance

To verify transferability across action representations, the authors evaluated DURA on $\pi_0$-FAST, an architecture that utilizes continuous action chunks rather than discrete tokens. As shown in Table 2, DURA achieved 100.0% ASR across all four LIBERO suites under both white-box and black-box access. In comparison, target-only baselines like TMA (58.0% white-box average), TMA-NES (31.3% black-box average), and UPA-RFAS (26.0% black-box average) struggled.

Setting	Attack Method	Supervision / Threat Model	Spatial ASR (%)	Object ASR (%)	Goal ASR (%)	Long ASR (%)	Average ASR (%)
Baseline	Benign (No Patch)	None	12.0	9.0	6.0	31.0	14.5
Baseline	Clean Patch	None	21.8	9.2	10.8	41.4	20.8
Simulated	TMA	White-Box (Target-Only)	55.0	46.0	78.0	53.0	58.0
Simulated	DURA (Ours)	White-Box (Target-Only)	100.0	100.0	100.0	100.0	100.0
Simulated	TMA-NES	Black-Box (Target-Only)	31.0	14.0	34.0	46.1	31.3
Simulated	UPA-RFAS	Black-Box (Target-Only)	27.0	14.0	21.0	42.0	26.0
Simulated	DURA (Ours)	Black-Box (Target-Only)	100.0	100.0	100.0	100.0	100.0

Attack Precision (AP) and Targeted Behavior Steering

A critical distinction highlighted by Han et al. is distinguishing true targeted policy control from incidental out-of-distribution policy degradation. To quantify this, the authors evaluated Attack Precision (AP) at the action-step level across $N$ rollout trajectories: $\text{AP} = \frac{1}{N} \sum_{i=1}^{N} \frac{T^{(i)}_{\text{target}}}{T^{(i)}}$ where $T^{(i)}$ is the total number of action steps in trajectory $i$, and $T^{(i)}_{\text{target}}$ is the number of steps where the robot’s output strictly matches the specified target behavior (e.g., action freezing or uncommanded forward motion). DURA achieved consistently high AP scores alongside high ASR, proving that the victim robot is actively forced along the attacker’s designated trajectory rather than merely experiencing random physical collapse.

---

5. In-Depth Technical Analyses: Stealthiness, Efficiency, and Physical Deployment

Visual Naturalness Metrics

Han et al. evaluated visual quality across six quantitative metrics: NIQE, CLIP-Natural, DISTS, SSIM, Boundary Seam Energy (BSE), and Patch Total Variation (Patch TV).

   NIQE (↓)         CLIP-Natural (↑)        DISTS (↓)           SSIM (↑)            BSE (↓)          Patch TV (↓)
[Lower is better]   [Higher is better]  [Lower is better]   [Higher is better]  [Lower is better]   [Lower is better]
  DURA ≈ Clean        DURA ≈ Clean        DURA ≈ Clean        DURA ≈ Clean        DURA ≈ Clean        DURA ≈ Clean
 Pixel Attacks High  Pixel Attacks Low   Pixel Attacks High  Pixel Attacks Low   Pixel Attacks High  Pixel Attacks High

Compared to direct pixel-space baselines (UMA, UPA, UADA), which introduce high-frequency noise and prominent boundary seams, DURA patches consistently scored closest to the benign “no-patch” reference. By operating within the VAE latent space of a pretrained diffusion model, DURA generates smooth, naturalistic textures that align with surrounding physical priors.

Single-GPU Wall-Clock Time Efficiency

To evaluate practical compute requirements, the authors benchmarked optimization time on a single NVIDIA H200 GPU with batch size 16:

• White-Box Convergence: DURA converges to high ASR within minutes by taking efficient DDIM steps along the latent trajectory. In contrast, pixel-space baselines (TMA, UPA, UADA) require multiple hours of per-instance pixel optimization to approach comparable success rates.
• Black-Box Scalability: DURA scales efficiently over time, rapidly outperforming query-based baselines. While TMA-NES plateaus at low success rates even when granted extended runtimes exceeding 20 hours, DURA reaches near-peak ASR within a fraction of the compute time.

Query Efficiency & Budget

The authors analyzed the effect of the Monte Carlo query budget $K$ per update in black-box settings ($128 \le K \le 2048$). DURA achieves strong attack performance at $K = 512$ queries per update and reaches saturation (100% ASR) at $K = 2048$. Conversely, TMA-NES plateaus between 40% and 48% ASR across identical query allocations. Searching inside a diffusion-guided latent space allows DURA to extract significantly more directional information per query than direct pixel-space search.

Patch Size Sensitivity & Variance

A footprint study on OpenVLA measured attack success relative to the percentage of total image area occupied by the patch:

• $\le 1\%$ Area: Modest attack success.
• $2\%$ Area: ASR increases sharply to $77\%$.
• $\ge 5\%$ Area: ASR reaches $99\%$ and saturates at $100\%$.

Crucially, the authors observed that seed-to-seed variance peaks in the 2%–3% transition zone and drops to zero once the footprint reaches $\ge 5\%$ area, where all random seeds achieve $100\%$ ASR.

Physical World Deployment (Franka Arm)

To validate real-world physical transferability, the authors printed optimized patches and placed them inside the camera workspace of a 7-DoF Franka robot arm executing a “bread-in-basket” manipulation task.

Without altering language instructions, policy weights, or robot hardware setups, the physical presence of the printed patch successfully hijacked execution. The patch induced targeted behaviors—such as action freezing or uncommanded forward motion—under real-world imaging conditions, including printing variations, ambient lighting shifts, camera perspectives, and physical placement noise.

---

6. Key Takeaways & Defense Implications

• Unrestricted Latent Attacks: Searching along diffusion latent trajectories generates visually natural, realistic patches that avoid conspicuous high-frequency noise artifacts while maintaining high attack effectiveness.
• Action-Only Vulnerability: Black-box VLA deployments can be manipulated using predicted action outputs alone via score-function estimation, requiring zero parameter or gradient access.
• Cross-Architecture Applicability: DURA successfully hijacks both discrete token-based policies (OpenVLA) and continuous action-chunking architectures ($\pi_0$-FAST) with up to 100% success rates.
• Physical Transferability: Adversarial perturbations optimized in diffusion latent spaces transfer to printed, physical patches that operate reliably under real-world lighting, perspective shifts, and placement noise.
• Need for Naturalistic Red-Teaming: VLA safety defenses must move beyond detecting synthetic, high-frequency pixel noise and integrate naturalistic physical scene perturbations into automated CI/CD red-teaming pipelines.

Han et al.’s findings demonstrate that physical workspace content itself can function as an attack surface for embodied AI systems. Consequently, robotic defenses must incorporate diffusion-guided, physically placeable scene perturbations into standard evaluation suites before deploying general-purpose VLA models into safety-critical human environments.

Read the full paper on arXiv · PDF