FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement

1. Introduction: The Loop of Insanity in Embodied AI

A recurring failure mode in robot policies trained on offline demonstrations is the tendency to repeat identical mistakes when encountering out-of-distribution (OOD) states. Because these policies are trained to imitate expert behavior, they lack the internal mechanisms to correct a trajectory once it deviates from the training distribution. When a robot misses a grasp or drops an object, simply re-executing the policy—a “naive retry”—frequently results in the system performing the same unsuccessful action repeatedly. This occurs because the robot remains stuck in a state that the offline data cannot resolve, leading to a feedback loop of identical failures.

In the paper “FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement” (2026), Hao et al. introduce the Failure-Aware Retry (FAR) framework to address this lack of adaptability. The core objective of the FAR framework is to enable autonomous adaptation at test-time, allowing robots to learn from immediate failures and adjust their behavior in situ without requiring human intervention or environment resets.

2. The Mechanics of Failure-Aware Retry (FAR)

The authors utilize a two-pronged approach to improve recovery, combining immediate policy weight updates with structured exploration.

2.1 Failure-Contrastive Preference Adaptation (FCPA)

Hao et al. propose Failure-Contrastive Preference Adaptation (FCPA) to steer the policy away from unsuccessful behaviors. The framework identifies failure-inducing action chunks using value estimation (Q and V functions). Once a failure is detected, the system constructs contrastive pairs. These consist of the unsuccessful action (the negative sample) and alternative candidate actions sampled from the current policy that have higher estimated values (the positive samples). By optimizing a preference objective, the policy is updated to assign lower denoising error to the preferred actions, effectively suppressing the failed behavior in subsequent retries.

2.2 Structured Exploration via Perturbation

To prevent the robot from remaining trapped in OOD states where the offline policy may have no valid samples, the authors detail a method for structured exploration. This involves injecting lightweight Gaussian perturbations into actions. To ensure mechanical stability on real-world hardware, the authors apply exponential smoothing to the noise, maintaining stable execution while encouraging local exploration of the state space.

Feature	Naive Retry (Re-execution)	Failure-Aware Retry (FAR)
Adaptation Mechanism	None; repeats the same action sequence.	FCPA; performs 5–10 gradient steps at test-time.
Exploration Strategy	Limited to inherent policy stochasticity.	Structured Gaussian perturbations with exponential smoothing.
Learning Signal	None.	Contrastive preference pairs based on $V(s)$ drops.
Recovery Strategy	Non-parametric (sampling-based).	Parametric (policy weight updates).

3. Attributing Failure: The Role of Value Estimation

Effective test-time adaptation requires identifying the exact actions responsible for a failure. Hao et al. employ Implicit Q-Learning (IQL) to provide conservative value estimates. This is critical because standard critics often overestimate values in OOD states. The authors utilize an expectile regression loss for the value function, defined as: $L_V = \mathbb{E}_{(s,a)\sim\mathcal{D}} [L_\tau^2(Q_\phi(s,a) - V_\psi(s))]$ This objective ensures conservatism by learning values primarily from actions supported by the offline data, providing a more reliable signal for failure detection.

The framework identifies failure-inducing behavior by measuring the value drop ($\Delta V_t$) across action chunks. A successful trajectory is characterized by a steady increase in the value function $V(s_t)$, representing progress toward the goal. Conversely, a failure is identified by a “pronounced decrease in value.” The authors’ data indicates that while a success plot shows a consistent upward trend, a failure plot reveals a sharp dip, which serves as the quantitative signal for intervention. To isolate negative samples, the framework ranks all chunks by $\Delta V_t$ and selects the lowest $\rho$-percentile, ensuring that only the most significant deviations are used for policy suppression.

4. From Recovery to Robustness: Continual Policy Improvement

Successful recovery trajectories are utilized beyond the immediate task to improve long-term robustness. The authors maintain three replay buffers to fine-tune the policy online:

• $\mathcal{D}_{exp}$: Offline expert demonstrations.
• $\mathcal{D}_{succ}$: Successful online trajectories, including recovery successes.
• $\mathcal{D}_{fail}$: Trajectories resulting in failure.

To prevent the policy from imitating suboptimal or “stagnant” movements, Hao et al. implement an Advantage-weighted Policy Update. The actor is updated only on expert and successful data ($\mathcal{D}_{exp} \cup \mathcal{D}_{succ}$), using the weight $w(s,a) = \exp(A(s,a)/\eta)$, where $A(s,a) = Q_\phi(s,a) - V_\psi(s)$. This weight acts as a filter; by prioritizing actions with high advantage, the policy is prevented from learning the low-value, repetitive behaviors characteristic of a naive retry.

5. Empirical Performance: Simulation and Real-World Results

Hao et al. report that FAR achieves a 17.6% average improvement over standard diffusion policies in ManiSkill, RoboSuite, and RoboMimic simulations. In real-world xArm manipulation tasks, the framework achieved an 11.7% gain. Notably, the authors contrast FAR with Bellman-Guided Retrials (DP-BGR), a non-parametric baseline that uses rejection sampling to avoid repetition. While BGR provides a baseline for retries, it struggles with environment distribution shifts caused by earlier failed attempts—a problem FAR overcomes through its parametric test-time updates.

The authors evaluated the framework on three real-world tasks:

• Drawer: A shorter-horizon task where most methods, including standard baselines, performed well.
• Pot: A more complex task where the authors identify that recovery is significantly more difficult due to the coordination required.
• Tea: A long-horizon task where identifying the specific failure-inducing action via $\Delta V_t$ is critical for a successful retry.

6. Conclusion: Implications for Failure-First AI Safety

The FAR framework demonstrates that autonomous recovery reduces the need for costly environment resets and human supervision. For AI safety researchers, the research offers three critical takeaways:

1. Failures as Supervision: The authors demonstrate that standard data collection underutilizes “hard” failure cases. These cases provide critical boundary information regarding the policy’s current limits that is systematically absent from expert-only offline datasets. FAR transforms these boundaries into a dense learning signal.
2. Test-Time Adaptation: The paper suggests that 5–10 gradient steps in the field can be more effective for recovery than massive amounts of offline pre-training, as it allows the model to adapt to the specific distribution shifts of the current environment.
3. Autonomous Recovery: By closing the loop between value-based failure detection and policy adjustment, robot systems can self-correct in complex environments.

The authors note that the framework currently relies on external environment feedback or time limits for failure detection. They suggest that future integration with Vision-Language Models (VLMs) could provide a more principled, autonomous mechanism for failure reasoning and detection.

Read the full paper on arXiv · PDF