RoboStriker: Latent-Space Strategic Games for Autonomous Humanoid Boxing

---

1. Introduction: The Embodied Cold-Start Problem

In multi-agent reinforcement learning (MARL) for humanoid robotics, researchers face a fundamental conflict between active strategic exploration and strict physical balance limits. As detailed by Yin et al., applying MARL directly to contact-rich, highly dynamic physical tasks—such as autonomous humanoid boxing—exposes a severe failure mode: unconstrained exploration in high-dimensional motor spaces leads to catastrophic joint-level physical collapse before viable combat tactics can emerge.

When a humanoid robot such as the Unitree G1 operates within an unconstrained, high-dimensional raw motor space ($29$-degree-of-freedom joint position targets, $A_i \subset \mathbb{R}^{29}$), the vast majority of arbitrary joint combinations correspond to physically invalid or unstable postures. Standard reinforcement learning algorithms rely on active, diverse exploration to discover high-reward maneuvers. However, in raw motor spaces, this unconstrained exploration triggers severe balance loss and mechanical collapse almost immediately. Agents are penalized heavily for these physical violations, creating an embodied cold-start problem where discovering even a single valid combat maneuver is mathematically and physically improbable.

This physical fragility is further exacerbated in multi-agent environments. An active, evolving opponent introduces continuous, unpredictable physical perturbations. Without structural constraints in the motor space to absorb and withstand these impacts, the multi-agent learning process collapses under the burden of basic balance maintenance, preventing the emergence of higher-level tactical behaviors.

To resolve this bottleneck, Yin et al. introduce RoboStriker, a framework that formulates humanoid combat as a two-player latent-space zero-sum Markov game. By mapping high-level strategic reasoning over a topologically bounded latent manifold to a decoder-reachable action space via a pretrained motion decoder, RoboStriker restricts strategic exploration to a physically safe domain, preventing joint-level collapse while enabling the co-evolution of competitive multi-agent strategies.

---

2. Architectural Overview: Decoupling Strategy from Execution

RoboStriker addresses the physical cold-start problem through a three-stage hierarchical architecture. The core design principle relies on decoupling high-level strategic reasoning from low-level motor execution.

The pipeline first establishes a low-level physical foundation by learning to track human motion data, distills this capability into a bounded latent motion space, and finally conducts multi-agent strategy evolution strictly over this latent manifold.

Stage	Name	Core Function	Primary Inputs & Outputs	Underlying Method
Stage I	Expertise Motion Tracker ($\pi_{\text{trk}}$)	Learns low-level whole-body control to track diverse humanoid boxing primitives.	In: Proprioception ($s_{\text{prop}}$), Privileged state ($s_{\text{priv}}$), Reference pose window ($s_{\text{ref}}$) Out: $29$-DoF joint position targets ($a_t$)	Deep RL on motion capture dataset ($\mathcal{D}_{\text{motion}}$) using tracking & regularization rewards ($r_{\text{trk}}$).
Stage II	Motion Encoding via Topological Latent Distillation	Distills motion tracking expertise into a compact, continuous, topologically bounded latent manifold ($Z$).	In: Observation ($o_t$), Proprioception ($s_{\text{prop}}$) Out: Latent code ($z_t \in \mathbb{R}^{32}$), Reconstructed action ($\hat{a}_t$)	Teacher-student distillation with Encoder ($E_\phi$), Decoder ($D_\psi$), and State-Conditioned Prior ($P_\xi$).
Stage III(a)	Behavioral Warm-Up	Bypasses reward sparsity and cold-start failures by learning basic striking skills against a passive opponent.	In: Egocentric goal state ($s_{\text{goal}}$), Behavioral prior ($P_\xi^\perp$) Out: Residual latent command ($\Delta z_t$)	Residual policy ($\pi_\theta$) regularized via Adversarial Motion Priors (AMP Discriminator $C$).
Stage III(b)	Latent-Space Neural Fictitious Self-Play (LS-NFSP)	Drives strategic co-evolution over the latent manifold without out-of-distribution physical actions.	In: Ego/opponent states, Rollout buffer ($\mathcal{D}_{\text{RL}}$), Reservoir buffer ($\mathcal{D}_{\text{SL}}$) Out: Mixed latent strategy ($\sigma_z$)	Dual-policy learning: Best-Response Policy ($\pi_z^{\text{RL}}$ via PPO) and Average Policy ($\bar{\pi}_z$ via SL).

---

3. Deconstructing the RoboStriker Pipeline

To establish a mathematical foundation for strategic co-evolution, Yin et al. formalize the humanoid combat task as a two-player latent-space zero-sum Markov game defined by the formal tuple:

$M = \langle I, S, \{O\}_i, \{A\}_i, \{Z\}_i, D_\psi, P, \{U\}_i, \gamma \rangle$

• Player Set ($I$): $I = \{1, 2\}$, representing the two competing humanoid agents.
• State Space ($S$): The global environment state space containing full physical dynamics and kinematics.
• Observation Spaces ($\{O\}_i$): Partial observations $O_i$ available to agent $i$.
• Raw Action Spaces ($\{A\}_i$): $A_i \subset \mathbb{R}^{29}$, representing raw joint position targets for the 29-DoF Unitree G1 humanoid.
• Continuous Latent Spaces ($\{Z\}_i$): $Z_i \subset \mathbb{R}^d$ ($d=32$), representing the bounded continuous latent manifolds over which strategic action selection takes place.
• Pretrained Decoder ($D_\psi$): $D_\psi: S \times Z_i \to A_i$, mapping state-conditioned latent decisions to executable motor commands on the decoder-reachable action manifold.
• Transition & Utility Functions ($P, \{U\}_i$): Environment dynamics $P^{\text{env}}: S \times A_1 \times A_2 \to \Delta(S)$ and raw utility $u_i^{\text{env}}(s, a_1, a_2)$ induce the latent transition function $P: S \times Z_1 \times Z_2 \to \Delta(S)$ and latent utility $u_i(s, z_1, z_2) = u_i^{\text{env}}(s, D_\psi(s, z_1), D_\psi(s, z_2))$, discounted by factor $\gamma \in [0, 1)$.

+-----------------------------------------------------------------------------------+
| STAGE I: Universal Motion Tracker (π_trk)                                         |
| Retargets Xsens MoCap data (30 min, 50 Hz) -> Unitree G1 29-DoF Joint Targets      |
+-----------------------------------------------------------------------------------+
                                          │
                                          ▼
+-----------------------------------------------------------------------------------+
| STAGE II: Topological Latent Distillation                                         |
| Encoder E_ϕ & Prior P_ξ -> Normalization -> Unit Hypersphere (z_t ∈ R^32)          |
| Pretrained Decoder D_ψ: S × Z -> A (29-DoF Motor Manifold)                       |
+-----------------------------------------------------------------------------------+
                                          │
                                          ▼
+-----------------------------------------------------------------------------------+
| STAGE III(a): Behavioral Warm-Up (AMP Curriculum)                                 |
| Residual Policy π_θ (Δz_t) over Prior P_ξ^⊥ -> Output z_t = Normalize(Δz_t + z_t^p) |
| AMP Discriminator C provides style reward r_style against MoCap distribution      |
+-----------------------------------------------------------------------------------+
                                          │
                                          ▼
+-----------------------------------------------------------------------------------+
| STAGE III(b): Latent-Space Neural Fictitious Self-Play (LS-NFSP)                  |
| Strategic Game M over Latent Space Z                                              |
|  ├─ Best-Response Policy π_z^RL  <-- PPO on Rollout Buffer D_RL                    |
|  ├─ Average Policy π̄_z           <-- Supervised Learning on Reservoir Buffer D_SL |
|  └─ Mixed Action Selection: σ_z = η · π̄_z + (1 - η) · π_z^RL (η = 0.1)            |
+-----------------------------------------------------------------------------------+

3.1 Stage I: Learning the Universal Motion Tracker

To establish a stable motor foundation, the authors collect approximately 30 minutes of boxing motion capture data from professional boxers using an Xsens motion capture system at 50 Hz. This dataset is doubled via left-right mirroring to build $\mathcal{D}_{\text{motion}}$, covering strikes, defenses, footwork, and transitions. Captured motions are retargeted to the 29-degree-of-freedom Unitree G1 morphology using Generalized Motion Retargeting.

The learned tracking policy $\pi_{\text{trk}}(a_t \mid s_t^{\text{self}}, s_t^{\text{ref}})$ receives:

• State $s_t^{\text{self}} = (s_t^{\text{prop}}, s_t^{\text{priv}})$, where $s_t^{\text{prop}}$ represents onboard proprioception and $s_t^{\text{priv}}$ contains privileged environment features.
• Reference motion goal $s_t^{\text{ref}}$, consisting of a short-horizon pose window of $K=3$ future poses extracted from $\mathcal{D}_{\text{motion}}$.
• Previous action $a_{t-1}$.

The tracking reward $r_{\text{trk}}$ combines exponential terms for root and body pose tracking, orientation matching, linear and angular velocity alignment, and penalty terms covering action rate, joint limits, and undesired contact forces. Early episode termination occurs upon pose collapse, end-effector height violations, or reaching a 10-second limit.

3.2 Stage II: Topological Latent Distillation

Stage II distills the tracking policy’s expertise into a compact, continuous latent space $\mathcal{Z} \subset \mathbb{R}^d$ ($d=32$). The framework uses a teacher-student setup:

1. Encoder ($E_\phi$): Maps full tracking observations $o_t = (s_t^{\text{prop}}, s_t^{\text{ref}})$ to a diagonal Gaussian distribution $E_\phi(z_t \mid o_t) = \mathcal{N}(z_t \mid \mu_e, \sigma_e)$.
2. Decoder ($D_\psi$): Reconstructs the target action $\hat{a}_t \sim D_\psi(\cdot \mid s_t^{\text{prop}}, z_t)$ to match the tracking teacher’s action $a_t \sim \pi_{\text{trk}}(\cdot \mid o_t)$.
3. State-Conditioned Prior ($P_\xi$): Models valid transitions from the current state using proprioception alone, parameterized as $P_\xi(z_t \mid s_t^{\text{prop}}) = \mathcal{N}(z_t \mid \mu_p, \sigma_p)$.

To bound the strategic action space, latent vectors are explicitly normalized onto a unit hypersphere:

$\hat{z} = \text{Normalize}(z) = \frac{E_\phi(o)}{\|E_\phi(o)\|_2}$

The student model optimizes a joint loss function $L_{\text{distill}} = L_{\text{rec}} + \lambda_{\text{prior}} L_{\text{prior}}$ with $\lambda_{\text{prior}} = 0.001$:

• Reconstruction Loss: $L_{\text{rec}} = \mathbb{E}[\|a_t - \hat{a}_t\|^2]$
• Prior Alignment Loss: $L_{\text{prior}} = \mathbb{E}\left[ D_{\text{KL}}\left( E_\phi(z_t \mid o_t) \parallel P_\xi(z_t \mid s_t^{\text{prop}}) \right) \right]$

The Kullback-Leibler (KL) divergence term regularizes the latent space toward the state-conditioned prior, preventing posterior collapse and ensuring that all latent vectors map through $D_\psi$ to physically valid motor states.

3.3 Stage III(a): Behavioral Warmup with Adversarial Motion Priors (AMP)

Initiating self-play directly leads to severe non-stationarity and reward sparsity. Stage III(a) introduces a behavioral warmup phase against a stationary target robot maintaining a standing stance.

A learnable residual policy $\pi_\theta(\cdot \mid s_t^{\text{goal}})$ outputs residual latent commands $\Delta z_t$ over a fixed behavioral prior $P_\xi^\perp$. The final latent decision is generated via normalized residual addition:

$z_t = \text{Normalize}(\Delta z_t + z_t^p)$

where $z_t^p \sim P_\xi^\perp(\cdot \mid s_t^{\text{prop}})$. Crucially, the goal state $s_t^{\text{goal}}$ concatenates ego-centric relative geometric vectors between the ego robot’s fists and the opponent’s torso, and between the opponent’s fists and the ego torso. These relative position vectors are explicitly rotated into the ego root frame so that striking and threat cues remain strictly orientation-invariant regardless of global arena orientation.

To maintain physical authenticity during strategic adaptation, an Adversarial Motion Prior (AMP) discriminator $C$ evaluates motion transitions produced by $(\pi_z, D_\psi)$ against reference motion clips from $\mathcal{D}_{\text{motion}}$. The discriminator processes local joint rotations, joint velocities, and base angular velocity, incorporating a gradient penalty $w_{\text{gp}}$ on real motion transitions:

$\min_C \mathbb{E}_{\mathcal{D}_{\text{motion}}}\left[(C(o_t^{\text{disc}}, o_{t+1}^{\text{disc}}) - 1)^2\right] + \mathbb{E}_{\pi_z, D_\psi}\left[(C(o_t^{\text{disc}}, o_{t+1}^{\text{disc}}) + 1)^2\right] + w_{\text{gp}} \mathbb{E}_{\mathcal{D}_{\text{motion}}}\left[\|\nabla C(o_t^{\text{disc}}, o_{t+1}^{\text{disc}})\|^2\right]$

The discriminator generates a style reward $r_{\text{style}}$, which is combined with task performance rewards (facing alignment, approach velocity, striking distance, contact-validated hits) into an integrated optimization target:

$R(s_t, z_t) = w_{\text{task}} \cdot r_{\text{task}}(s_t) + w_{\text{style}} \cdot r_{\text{style}}(s_t, s_{t+1})$

3.4 Stage III(b): Latent-Space Neural Fictitious Self-Play (LS-NFSP)

Building on the warmup policy, competitive multi-agent co-evolution is driven by Latent-Space Neural Fictitious Self-Play (LS-NFSP). Running self-play over raw motor actions causes policy collapse due to non-transitive policy cycling and extreme joint instability. LS-NFSP constrains strategy updates entirely to the unit hypersphere $\mathcal{Z} \subset \mathbb{R}^{32}$.

Each agent maintains a dual-policy architecture:

1. Best-Response Policy ($\pi_z^{\text{RL}}$): Trained via Proximal Policy Optimization (PPO) using recent experience stored in a rollout buffer $\mathcal{D}_{\text{RL}}$.
2. Average Policy ($\bar{\pi}_z$): Trained via supervised learning to imitate historical best-response behaviors stored in a reservoir buffer $\mathcal{D}_{\text{SL}}$.

The reservoir buffer $\mathcal{D}_{\text{SL}}$ maintains a capacity of $K = 10^6$ samples per agent, updated using reservoir sampling (where step $n > K$ replaces an existing entry with probability $K/n$) to prevent catastrophic forgetting. Action selection follows a mixed strategy governed by an anticipatory parameter $\eta = 0.1$:

$\sigma_z = \eta \cdot \bar{\pi}_z + (1 - \eta) \cdot \pi_z^{\text{RL}}$

Theoretical Game Justification: Under standard regularity assumptions, treating the PPO policy updates as bounded-error approximate best responses to historical opponent strategy distributions induces an $\epsilon$-Nash equilibrium over the decoder-reachable motor manifold. Because optimization is confined to the topologically bounded latent manifold $\mathcal{Z}$, policy updates cannot select unconstrained, out-of-distribution physical actions. This theoretical property guarantees bounded policy convergence, suppressing strategy cycling and maintaining stability during adversarial interactions.

---

4. Empirical Evaluation & Tactical Performance Comparison

Yin et al. evaluate RoboStriker on $29$-DoF Unitree G1 humanoids within Isaac Lab, running $4,096$ parallel environments. Methods are evaluated across tactical proficiency and physical stability dimensions:

• Win Rate (%): Percentage of evaluation bouts won (a bout terminates when any non-foot body part touches the ground).
• Offensive Landing Rate ($\eta_{\text{hit}}$): Proportion of offensive strikes exceeding $F_{\text{th}} = 10\text{ N}$ contact force.
• Engagement Rate ($ER$): Temporal fraction during which the agent maintains effective striking distance $d_t \in [0.5, 1.2]\text{ m}$ and facing alignment $\cos(\theta_t) > 0.9$.
• Base Orientation Stability ($BOS$): Measures upright posture maintenance under perturbations via gravity vector angular deviation: $BOS = \mathbb{E}[\exp(-\|g_{\text{base}} - g_{\text{world}}\|^2)]$. Higher is better.
• Torque Smoothness ($TS_\tau$): Quantifies high-frequency motor oscillations and actuator chatter as the mean temporal torque delta: $TS_\tau = \mathbb{E}[\|\tau_t - \tau_{t-1}\|]$. Lower is better.

The table below summarizes the quantitative evaluation metrics reported in Table I of the paper:

Methods	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	$\eta_{\text{hit}} \uparrow$	$ER \uparrow$
(1) RoboStriker	–	68.52	76.24	82.41	84.47	95.38	98.50	100.0	92.50	0.685 $\pm$ 0.03	0.824 $\pm$ 0.02
(2) Fictitious SP	–	–	62.35	71.84	75.16	92.11	96.18	99.42	88.14	0.420 $\pm$ 0.03	0.650 $\pm$ 0.04
(3) Naive SP	–	–	–	68.45	70.38	88.61	94.52	98.53	85.25	0.350 $\pm$ 0.04	0.580 $\pm$ 0.05
(4) RoboStriker w/o AMP	–	–	–	–	62.12	82.14	88.37	95.18	78.40	0.490 $\pm$ 0.03	0.720 $\pm$ 0.05
(5) PPO-Only	–	–	–	–	–	45.24	77.86	82.11	68.55	0.231 $\pm$ 0.03	0.495 $\pm$ 0.02
(6) Static-Target Specialist	–	–	–	–	–	–	60.33	85.58	62.18	0.210 $\pm$ 0.04	0.450 $\pm$ 0.06
(7) RoboStriker w/o Warmup	–	–	–	–	–	–	–	94.17	51.34	0.050 $\pm$ 0.02	0.120 $\pm$ 0.05
(8) 29-DoF Action-Space SP	–	–	–	–	–	–	–	–	11.50	0.142 $\pm$ 0.05	0.315 $\pm$ 0.08
(9) 29-DoF Action-Space + AMP	–	–	–	–	–	–	–	–	–	0.285 $\pm$ 0.04	0.465 $\pm$ 0.05

Analytical Synthesis of Results & Physical Stability

• Complete Collapse of Unconstrained Motor Exploration: Direct exploration in the raw joint space (29-DoF Action-Space SP) fails entirely, yielding a $0.0\%$ win rate against RoboStriker, an offensive landing rate of $0.142$, and an engagement rate of $0.315$. Agents suffer immediate balance loss upon executing unconstrained exploration steps, reflected in low Base Orientation Stability ($BOS$) and severe torque noise ($TS_\tau$).
• Physical Stability Breakdown ($BOS$ and $TS_\tau$ Analysis): As demonstrated in Figure 2 of the source text, 29-DoF Action-Space SP exhibits rapid degradation in $BOS$ due to frequent postural falls and high $TS_\tau$ chatter ($TS_\tau > 6.0\text{ N}\cdot\text{m}$). Adding AMP (29-DoF Action-Space + AMP) regularizes joint transitions, lowering torque noise and raising $BOS$. However, RoboStriker achieves superior posture maintenance ($BOS \approx 0.92$) and minimal torque chatter ($TS_\tau \approx 1.8\text{ N}\cdot\text{m}$).
• Latent Abstraction vs. Motion Regularization: Adding AMP to raw motor exploration improves stability ($\eta_{\text{hit}}$ increases from $0.142$ to $0.285$; $ER$ from $0.315$ to $0.465$). However, RoboStriker still defeats this baseline in $92.50\%$ of cross-play bouts while achieving significantly higher striking efficiency ($\eta_{\text{hit}} = 0.685$). This proves that behavioral style regularization (AMP) alone cannot replace the strategic search structure provided by a bounded latent action manifold.
• Baseline Failure Modes:
  • PPO-Only agents exhibit passive behaviors due to the lack of competitive co-evolution, yielding low engagement ($ER = 0.495$) and failing to discover offensive tactics ($\eta_{\text{hit}} = 0.231$).
  • Naive SP exhibits severe cyclical performance oscillations caused by policy forgetting and non-transitive strategy loops, limiting engagement to $0.580$.
  • RoboStriker w/o Warmup fails due to severe reward sparsity, with tactical performance flatlining near zero ($\eta_{\text{hit}} = 0.050$).

---

5. Parameter Ablations and Hardware Deployment

5.1 Latent Dimension Trade-off Analysis

The authors conduct ablation experiments to determine the optimal latent space dimensionality ($d$). The performance metrics from Table II of the source text are structured below:

Latent Dimension	Kinematic Error: MPKPE (mm) $\downarrow$	Win Rate vs. 32D (%)	Offensive Landing Rate ($\eta_{\text{hit}}$) $\uparrow$
8D	$128.2 \pm 4.4$	7.1%	$0.328 \pm 0.03$
16D	$85.9 \pm 1.7$	23.6%	$0.334 \pm 0.03$
32D (RoboStriker)	$38.5 \pm 0.4$	–	$0.685 \pm 0.03$
64D	$35.0 \pm 0.4$	39.2%	$0.520 \pm 0.08$

• Dimensionality Analysis: At low dimensionalities ($8\text{D}$ and $16\text{D}$), the latent space lacks the expressiveness required to encode complex motions, resulting in high Mean Per Keypoint Position Error ($\text{MPKPE} = 128.2\text{ mm}$ for $8\text{D}$) and rigid, unnatural behavior. Expanding to $64\text{D}$ yields minimal kinematic tracking gains ($35.0\text{ mm}$ vs $38.5\text{ mm}$) but overcomplicates high-level exploration. The added manifold redundancy slows multi-agent convergence, reducing the offensive landing rate from $0.685$ down to $0.520$. Thus, $d=32$ offers the optimal trade-off between kinematic fidelity and multi-agent search efficiency.

5.2 Physical Sim-to-Real Hardware Transfer

To validate physical viability, the learned policies were transferred from simulation (Isaac Lab on NVIDIA RTX 4090 GPUs) to physical 29-DoF Unitree G1 humanoid hardware.

• Control Infrastructure: Policies execute at 50 Hz using ONNX Runtime on the robots’ onboard CPUs. Inter-robot state communication is managed via ROS 2. Motion-capture pelvis markers tracked at 100 Hz supply real-time relative spatial coordinates, which are combined with onboard proprioception and forward kinematics to form the goal observation state $s_{\text{goal}}$.
• Domain Randomization & Gain Scaling Precision: Training incorporates domain randomization over contact friction coefficients, joint position biases, torso center-of-mass offsets, and intermittent external push forces. To prevent high-frequency actuator chatter and torque discontinuities, joint PD gains are scaled proportional to joint inertia ($I_j$) according to explicit natural frequency and damping parameters:

$k_{p,j} = I_j \omega^2, \quad k_{d,j} = 2 I_j \zeta \omega$

where the natural frequency is set to $\omega = 10\text{ Hz}$ and the damping ratio is set to $\zeta = 2$. Robots are initialized using a high-stiffness standing controller before smoothly transitioning to dynamic latent-space policy control. Real-world hardware trials confirmed that robots executed stable, contact-rich maneuvers without physical balance collapse.

---

6. Key Takeaways & Safety Implications

• Representation Constraints as Physical Safety Boundaries: Restricting high-level reinforcement learning exploration to a learned, topologically bounded latent manifold ($\mathcal{Z} \subset \mathbb{R}^{32}$) acts as an effective safety bound. It prevents physical agents from sampling invalid, high-risk joint target combinations, suppressing mechanical balance collapse during exploration in real-world embodied deployments.
• Decoupled Co-Evolution: Separating game-theoretic strategy evolution (LS-NFSP) from low-level motor execution (expertise tracker $\pi_{\text{trk}}$) maintains operational balance stability. This separation allows agents to learn counter-strategies against non-stationary opponents without jeopardizing basic locomotion integrity.
• Complementarity of AMP and Latent Space: Style regularization (AMP) and geometric latent interfaces address distinct aspects of agent control. AMP maintains motion regularity relative to human demonstrations, while latent action abstraction structures the search space for high-level multi-agent optimization.

Read the full paper on arXiv · PDF