Action ControlNet: A Lightweight Delay-Aware Adapter for Smooth Asynchronous Control in Vision-Language-Action Models

Focus: Vision-language-action (VLA) models promise general-purpose robot manipulation, but their inference latency is a persistent obstacle to stable high-frequency control. Asynchronous execution overlaps inference with execution to hide that latency, yet the next action chunk is still predicted from observations that are already stale by the time the chunk is emitted — while the robot has continued to move. That mismatch produces handoff discontinuities, action jitter, and failures specifically in contact-rich manipulation, where small timing errors compound. The paper introduces Action ControlNet (ACNet), a lightweight delay-aware adapter that uses the already-executed motion suffix as a residual condition for a mostly frozen action head, rather than retraining the policy under delay.

Key Insights

• The failure is latent and timing-induced, not perceptual. The model’s perception and planning are correct; the failure arises from the gap between observation time and action-emission time. This is a deployment failure mode that only surfaces under real-time execution, where it is invisible to offline, single-step evaluations that feed the policy fresh observations.
• Mitigation without retraining the backbone. ACNet leaves the pretrained VLA backbone unchanged, adds few trainable parameters, and stays compatible with generative action heads (diffusion and flow matching). It treats delay-robustness as a residual adaptation problem on top of a frozen policy, which is cheaper than full delay-conditioned retraining and preserves prior capabilities.
• Empirical scope. Evaluated on Kinetix, Meta-World MT50, and a real-world SO-ARM101 platform, ACNet improves robustness under inference delay and yields smoother asynchronous trajectories than direct chunk stitching. The contact-rich and real-platform settings are where the stale-observation failure mode matters most; the benchmarks that would not show it (single-step, quasi-static) are deliberately not the only evidence.

Failure-First Relevance

This is a failure-first paper in the literal sense: the object of study is a failure mode — stale-observation action chunks degrading contact-rich manipulation — rather than a capability claim. It fits the embodied-action failure boundary the repo draws between chat-LLM jailbreaks and VLA action failures: a VLA is trained to act, not to refuse, so its failures are physical-action failures (jitter, discontinuity, contact mistiming), not content refusals. The unsafe_action_elicitation metric class exists precisely for this category — the boundary where there is no refusal floor to break and the question is whether the emitted trajectory is safe. Latency-induced jitter is not itself an adversarial elicitation, but it is the same action-emission substrate an adversarial instruction would exploit, and it is exactly the kind of uncommanded motion the proposed HANSE Layer-4 Kinematic Shield is designed to catch. The paper’s contribution is also methodologically aligned: it isolates a single failure mechanism (stale observations under async handoff), reproduces it, and mitigates it with the smallest intervention that closes the gap — the failure-first pattern of characterising the failure before reaching for the defence.