Stubborn: A Streamlined and Unified Reinforcement Learning Framework for Robust Motion Tracking and Fall Recovery for Humanoids

1. Introduction: The Fragility of Humanoid Locomotion

The public is frequently presented with polished videos of humanoid robots performing complex dances or acrobatic maneuvers. However, as robotics safety specialists recognize, these demonstrations often mask an underlying fragility: many humanoid control systems remain highly sensitive to external disturbances. When pushed or tripped, these robots frequently experience catastrophic failures because their control policies are optimized for “perfect” tracking conditions rather than recovery and resilience.

Research by Xiao Ren et al. introduces the “Stubborn” framework to bridge this gap. Rather than treating motion tracking and fall recovery as two separate, often conflicting objectives, the authors propose a unified approach. By treating tracking failures not as terminal states but as “natural gateways” to a fall-state manifold, the “Stubborn” framework allows humanoids to maintain high-fidelity motion while autonomously recovering from significant physical perturbations. This article explores how specific architectural innovations—from probabilistic termination to asymmetric learning—enable a significant shift toward failure-resilient embodied AI.

2. The Problem with “Hard” Failures

The paper identifies a fundamental bottleneck in existing Reinforcement Learning (RL) methods for humanoid locomotion: the reliance on “hard termination.” Traditionally, training environments immediately stop an episode if a robot’s tracking error exceeds a threshold, a design intended to keep the agent near the reference motion.

The authors identify two primary failures resulting from this approach:

• Hard Termination Mechanisms: By stopping the episode the moment a robot begins to fall, the agent is prevented from exploring the “fallen” state. Consequently, it never learns the motor sequences required to stabilize a descent or transition from a fallen state back to a standing posture.
• Architectural Complexity: To compensate for hard termination, existing systems often rely on multi-stage training or separate, specialized recovery policies. The authors argue this increases complexity and creates conflicting gradients, where “tracking” and “recovery” logics interfere with one another.

Instead, the paper advocates for a “failure-first” perspective: allowing the robot to remain active after an error harnesses natural transitions to learn recovery behaviors spontaneously within a single policy.

3. Innovation I: Asymmetric Architecture and Yaw-Aligned Representation

To facilitate robust sim-to-real transfer, the authors utilize an asymmetric Actor-Critic architecture. In this setup, the Critic has access to “privileged information” from the simulation—such as ground-truth terrain data and exact contact forces—while the Actor relies solely on proprioceptive observations. This allows the value function to learn high-fidelity estimations of state quality during training, which the Actor can then execute using only its onboard sensors in real-world environments like the Unitree G1.

Furthermore, the paper implements a drift-invariant tracking representation to decouple global translation/yaw drift from local body coordination. The framework transitions from a standard World Frame ($W$) to a Yaw-Aligned Frame ($\Psi$). This is achieved by applying the inverse yaw rotation $R_Z(\psi)^T$ to the body pose: $\tilde{p}_i = R_Z(\psi)^T(p_i^W - p_{root}^W), \quad \tilde{R}_i = R_Z(\psi)^T R_i^W$ By stripping the global heading component, the policy ignores absolute rotation in the room—which may drift after a push—and focuses instead on local whole-body coordination. This ensures the robot does not exert excessive control effort trying to “correct” its global heading at the expense of its immediate balance.

4. Innovation II: Bernoulli-Based Probabilistic Termination

The core of the “Stubborn” framework is the replacement of deterministic stops with a probabilistic mechanism. When the robot exceeds specific error thresholds—$\theta_{pos} = 0.25$m for root height or $\theta_{quat} = \pi/2$ rad for orientation—it enters an unstable regime where its “survival” is determined by chance rather than an immediate reset.

The Math of Resilience

The framework models episode termination using a conditional Bernoulli distribution: $P(\tau_t = 1 | s_t) = p_{term}$ where $p_{term} = 0.005$ is triggered only when tracking errors exceed $\theta_{pos}$ or $\theta_{quat}$.

The authors set $p_{term} = 0.005$ to create an expected exploration window of 200 steps ($E[T] = 1/p_{term}$), providing approximately 4 seconds of physical recovery time at a 50Hz control frequency. This window is essential for mitigating Temporal Difference (TD) distortion. In standard RL, hard truncation creates a value function “cliff” where the agent cannot see the potential reward of states beyond the failure point. By allowing the episode to continue probabilistically, the “Stubborn” framework smooths this value estimation, encouraging the agent to discover emergent recovery behaviors.

5. Innovation III: Error-Driven Adaptive Sampling (AdpS)

In standard training, “hard-sample mining” typically relies on episode termination to identify difficult segments. However, under a probabilistic termination regime, episode length is no longer a reliable indicator of difficulty. To maintain training efficiency, the authors introduced a “Tracking-Error-Driven Strategy” using adaptive sampling weights ($w_t$).

The process utilizes the average keypoint tracking error ($\bar{e}$) and specific update operators:

1. Weight Increments: If $\bar{e}$ exceeds the success threshold ($\theta_{success} = 0.06$), the weight for that segment is increased ($+\Delta w_i$), forcing the robot to practice the difficult motion more frequently.
2. Local Decay ($\Phi_{att}$): If the robot successfully tracks a segment to its full duration ($T_{max}$), the weight is decreased to focus resources elsewhere.
3. Local Redistribution ($\Phi_{dist}$): If an episode has low error but is terminated early by the Bernoulli chance ($T < T_{max}$), the system redistributes weights toward the end of that segment. This ensures the robot specifically learns the transitions out of that state, preventing “blind spots” in the motion sequence.

6. Evaluating Performance: Simulation and Stress Tests

The framework was evaluated against several state-of-the-art (SOTA) baselines on the LAFAN1 dataset. The authors included a “From-scratch RL” baseline to isolate the benefits of their specific innovations.

Method	MPBPE (mm) ↓	MPJPE ($10^{-3}$ rad) ↓	MPJVE ($10^{-3}$ rad/s) ↓	$\Delta acc$ (rad/s²) ↓
HoloMotion	162.92	150.65	758.56	19.47
Any2Track	401.26	122.44	913.41	26.44
BFM-Zero	214.09	299.36	1100.88	17.07
From-scratch RL	62.68	115.43	887.87	27.75
Stubborn (Ours)	48.85	113.38	624.03	17.09

Metrics: MPBPE (Body Position), MPJPE (Joint Position), MPJVE (Joint Velocity), $\Delta acc$ (Acceleration Deviation).

Stubborn outperformed all baselines in position and velocity tracking. Notably, it achieved an acceleration deviation ($\Delta acc$) of 17.09, significantly smoother than standard RL (27.75). In “Push Recovery” tests under a severe 5 m/s perturbation, the “Stubborn” framework maintained a 100% success rate, whereas the version without probabilistic termination (w/o PT) dropped to 77.5%.

7. Deployment on the Unitree G1 Humanoid

The authors validated the policy on the 29-DoF Unitree G1 humanoid. Despite being trained in the IsaacLab simulator, the asymmetric architecture allowed for a seamless sim-to-real transition.

In physical environments, the G1 successfully demonstrated:

• High-Dynamic Maneuvers: Tracking flips and acrobatic sequences with high precision.
• Autonomous Recovery: The robot successfully re-established stable tracking after being subjected to falls and external pushes, standing up and continuing its motion sequence without manual reset.

8. Conclusion: Takeaways for Failure-Resilient AI

The “Stubborn” framework illustrates that robustness does not require secondary, complex recovery architectures. Instead, it emerges from how an agent is taught to perceive its orientation and how it is allowed to experience failure.

Critical takeaways for the robotics community include:

1. Unified Failure Handling: Treating recovery as an extension of the primary task creates cohesive motor behaviors and simplifies the training pipeline.
2. Strategic “Stubbornness”: A probabilistic window for failure (e.g., the 4-second window) allows the agent to discover recovery maneuvers that are mathematically invisible under hard termination.
3. Dynamic Hard-Sample Mining: In non-deterministic environments, error-driven sampling using redistribution operators ($\Phi_{dist}$) is essential to ensure the agent learns to handle difficult state transitions.

The authors indicate that future work will focus on further improving the smoothness of these recovery motions and extending the “Stubborn” framework to operate on highly irregular and uneven terrains.

Read the full paper on arXiv · PDF