Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation

Introduction: The Failure Modes of Flat VLA Policies

Recent advancements in Vision-Language-Action (VLA) models have enabled general-purpose robotic manipulation by unifying visual perception, language understanding, and action generation within single foundation architectures. However, as demonstrated by Kong et al. in “Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation”, standard VLA post-training paradigms predominantly optimize models as flat policies.

Flat VLA policies map visual observations and a static global task instruction directly to low-level motor actions across an entire execution trajectory. While this end-to-end mapping operates reliably for short-horizon primitives, it induces structural failure modes when deployed on multi-stage, long-horizon manipulation workflows:

• Lack of Stage Tracking: Flat policies maintain identical high-level task instructions regardless of the current execution phase, leaving stage-wise sub-goals implicit. Consequently, the agent cannot reliably identify its current execution phase or determine the immediate intermediate objective.
• Compound Error Accumulation: Minor trajectory deviations or motor errors accumulate over time without intermediate semantic anchors to reset policy execution. Lacking stage-aware subgoals for self-correction, flat policies experience compounding failures that degrade execution accuracy over long horizons.

From an AI safety perspective, the interface between high-level task specification and low-level action execution represents a critical failure seam. When flat VLA policies encounter multi-step tasks, the lack of explicit temporal decomposition causes structural vulnerabilities where minor visual or physical perturbations lead to execution drift or outright task failure.

To resolve these structural failure modes, Kong et al. introduce Hierarchical Robotic Control (HiRoC). HiRoC is a post-training framework that explicitly decouples high-level semantic task planning from low-level motor execution, providing dynamic sub-goal guidance and stage-tracking mechanisms across complex manipulation tasks.

---

Architectural Framework: Decoupling Task Planning from Action Execution

HiRoC structures robotic control into a two-tier operational hierarchy: a high-level Vision-Language Model (VLM) planner that decomposes complex instructions into subgoals, and a low-level VLA executor that generates motor actions conditioned on those subgoals.

Module	Pretrained Backbone	Primary Function	Training Methodology
High-Level Planner	Qwen2.5-VL-3B	Decomposes global task instructions into sequential, executable subgoals	Supervised Fine-Tuning (SFT) with LoRA on curated data ($D_{plan}$)
Low-Level Executor	OpenVLA-OFT	Predicts low-level action chunks conditioned on visual inputs and subgoals	Subgoal-conditioned SFT followed by Online Hierarchical RL Tuning

The high-level planner processes the global task instruction alongside multimodal visual observations to generate textual subgoals, accompanied by expected image coordinates for target manipulation objects. High-level replanning is performed every 20 policy calls.

The low-level executor receives these fine-grained textual subgoals along with real-time visual observations to predict continuous action chunks with a temporal horizon of $H=8$. Optimization of the executor is conducted using BF16 Fully Sharded Data Parallel (FSDP) with a learning rate of $2 \times 10^{-5}$.

---

Overcoming Distribution Misalignment: The Pre-RL Alignment Step

Kong et al. identify a critical bottleneck in hierarchical post-training termed Distribution Misalignment (DM), which occurs at the planner-executor operational seam.

The Cold-Start Issue and Execution Paralysis

Pretrained VLA executors are optimized to map visual inputs directly to full tasks conditioned on broad global instructions. When a high-level planner is introduced to emit localized subgoals, the executor receives conditioning inputs outside its pre-training distribution. Directly applying online Reinforcement Learning (RL) to an unaligned executor results in severe cold-start failures and execution paralysis, as the policy cannot interpret or follow fine-grained language subgoals.

Subgoal-Conditioned SFT Alignment

To bridge this distributional gap prior to online RL tuning, Kong et al. reorganize demonstration trajectories into a specialized executor alignment dataset ($D_{exe}$). Trajectories are partitioned into intermediate state-subgoal-action tuples $(s_i, l_i, a_i)$, where:

• $s_i$ represents the multimodal visual observation for the active subtask.
• $l_i$ denotes the textual subgoal emitted by the planner.
• $a_i = \langle a_1, \dots, a_m \rangle$ represents the ground-truth action chunk required to accomplish $l_i$.

The executor undergoes supervised fine-tuning on $D_{exe}$ by minimizing the cross-entropy loss over action tokens conditioned on intermediate subgoals:

$L_{SFT}^{exe}(w_e) = -\mathbb{E}_{(s_i, l_i, a_i) \sim D_{exe}} \left[ \frac{1}{H \cdot D} \sum_{t=1}^H \sum_{d=1}^D \log P_\pi (\cdot \mid s_i, l_i) \right]$

where $w_e$ represents the executor parameters, $H$ is the action horizon ($H=8$), and $D$ is the action dimension. This pre-RL alignment step endows the executor with baseline subgoal-following capabilities, preventing execution paralysis when RL fine-tuning commences.

Empirical Impact of DM Pre-Training

Rollout evaluations by the authors confirm that distribution alignment prior to online RL is mandatory for policy convergence:

• LIBERO-Spatial Suite: 94.8% final success rate with DM pre-training (starting from a 49.7% initial rollout success rate) vs. 70.2% without DM pre-training (starting from 33.6%).
• LIBERO-Goal Suite: 83.8% final success rate with DM pre-training (starting from a 56.5% initial rollout success rate) vs. 54.4% without DM pre-training (starting from 24.5%).

---

Hierarchical Policy Optimization: Combining Global and Local GRPO

Following pre-RL alignment, HiRoC optimizes the low-level executor through online environment interactions guided by the planner. During this phase, the high-level planner remains strictly frozen.

Kong et al. freeze the planner during low-level RL for two safety and stability reasons:

1. Eliminating “Moving-Target” Instability: Updating the planner continuously alters the output distribution of emitted subgoals, destroying the planner-executor alignment established during SFT.
2. Resolving Credit Assignment Ambiguity: Sparse trajectory rewards make it impossible to differentiate whether a failure stems from invalid high-level planning or faulty low-level execution. Freezing the planner isolates policy updates to motor execution.

       +-------------------------------------------------------+
       |               High-Level VLM Planner                  |
       |             (Qwen2.5-VL-3B, Frozen SFT)               |
       +---------------------------+---------------------------+
                                   |
                                   | Subgoal (l_t) [Every 20 steps]
                                   v
+----------------------------------+----------------------------------+
|                      Low-Level VLA Executor                         |
|                         (OpenVLA-OFT)                               |
+----------------------------------+----------------------------------+
                                   |
                                   | Action Chunk (a_t:t+H-1, H=8)
                                   v
       +-------------------------------------------------------+
       |                     Environment                       |
       +---------------------------+---------------------------+
                                   |
                                   | State (s_t), Rewards (r_t)
                                   v
       +-------------------------------------------------------+
       |             Hierarchical GRPO Advantage               |
       |    A_i = w_t * A_i^(task) + w_s * A_i^(sub)           |
       +-------------------------------------------------------+

Rollout Infrastructure

Online RL fine-tuning is executed using Group Relative Policy Optimization (GRPO). The sampling configuration utilizes 64 vectorized environments per worker, 16 rollout epochs per policy update, and a maximum trajectory horizon of 512 environment steps.

Mathematical Formulation of Advantage Signals

For each task instance, a group of $N=8$ trajectories is sampled. To balance macro task completion with micro subtask progress, the authors formulate a dual-level advantage function:

1. Global Task Advantage ($A^{task}_i$): Derived from terminal trajectory rewards $R_i = \sum_{t=1}^{T_i} r_{i,t}$, standardized across the group: $A^{task}_i = \frac{R_i - \mu_g}{\sigma_g + \epsilon}$ where $\mu_g$ and $\sigma_g$ are the group-wise mean and standard deviation of total trajectory rewards.

2. Local Subgoal Advantage ($A^{sub}_i$): Calculated from intermediate subtask progress scores $S_i$ recorded across planner-generated segments: $A^{sub}_i = \frac{S_i - \mu_s}{\sigma_s + \epsilon}$ where $\mu_s$ and $\sigma_s$ represent group-wise subtask metrics.

3. Adaptive Advantage Weighting: The normalized advantage signals are unified dynamically: $\tilde{A}_i = w_t A^{task}_i + w_s A^{sub}_i$ To guarantee convex combination normalization ($w_t + w_s = 1$), the adaptive weights $w_t$ and $w_s$ are defined based on average subgoals $K = \frac{1}{N}\sum_{i=1}^N K_i$ and episode trajectory step length $L$: $w_t = \frac{L}{K + L}, \quad w_s = \frac{K}{K + L}$

Optimization Objective

The executor policy ($\pi_\theta$) is updated using a PPO-style clipped objective with token-level binary masking:

$L_{actor}(\theta) = -\frac{1}{\sum_{i,t,j} m_{i,t,j}} \sum_{i,t,j} m_{i,t,j} \min \left[ \rho_{i,t,j}(\theta)\tilde{A}_i, \text{clip}\left(\rho_{i,t,j}(\theta), 1-\epsilon_\ell, 1+\epsilon_u\right)\tilde{A}_i \right]$

where $\rho_{i,t,j}(\theta) = \frac{\pi_\theta(a_{i,t,j} \mid o_{i,t}, l_{i,t}, a_{i,t,<j})}{\pi_{\theta_{old}}(a_{i,t,j} \mid o_{i,t}, l_{i,t}, a_{i,t,<j})}$ represents the action token probability ratio, and $m_{i,t,j} \in \{0, 1\}$ is a binary mask restricting updates strictly to valid action tokens.

---

Empirical Evaluation & Performance Analysis

Kong et al. evaluated HiRoC across the four benchmark suites of LIBERO, zero-shot perturbation suites in LIBERO-Plus, and physical sim-to-real transfer environments.

Benchmark Results on LIBERO

HiRoC demonstrates consistent improvements over flat VLA baselines, flat online RL post-training methods, and implicit world-model architectures.

Benchmark Suite	Baseline (OpenVLA)	Baseline (VLA-RL)	HiRoC	Absolute Performance Gain
LIBERO-Spatial	84.7%	90.2%	95.6%	+10.9% over OpenVLA
LIBERO-Object	88.4%	91.8%	96.0%	+7.6% over OpenVLA
LIBERO-Goal	53.7%	59.8%	84.4%	+30.7% over OpenVLA
LIBERO-Long	79.2%	82.2%	98.0%	+18.8% over OpenVLA
Average Success Rate	76.5%	81.0%	93.5%	+17.0% (Overall Gain)

Contextualization Against World Models and Full SFT

To evaluate the specific advantage of explicit hierarchical post-training, the authors contextualized HiRoC against additional state-of-the-art paradigms reported in the literature:

• Implicit World-Model Baselines: Models like World-Env (79.6% average success rate) and GRAPE (80.2% average success rate) utilize world models or implicit preference dynamics. While these models learn environment dynamics, they lack explicit temporal sub-goal decomposition. HiRoC outperforms World-Env by +13.9% and GRAPE by +13.3%, demonstrating that explicit language-level subgoal decomposition provides superior guidance compared to implicit visual dynamics.
• Full SFT Baselines: OpenVLA*-Full (utilizing action chunking and parallel decoding) achieves a 91.0% average success rate. HiRoC outperforms this full SFT baseline by +2.5% overall, with marked dominance on long-horizon tasks (LIBERO-Long: 98.0% vs. 86.5%), confirming that hierarchical online RL interaction enables policies to surpass static demonstration boundaries.

Generalization and Sim-to-Real Transfer

• Zero-Shot Robustness (LIBERO-Plus): Evaluated across 7 distinct environmental perturbations (such as visual lighting changes, layout offsets, and object shifts), HiRoC maintained higher zero-shot task success rates than OpenVLA, OpenVLA*-One, and WorldVLA. Subgoal decomposition isolates low-level execution from global scene perturbations by bounding motor scope to fine-grained visual subgoals.
• Physical Hardware Deployment: Using JoyRA-0.1 as the base VLA model inside JoySim, HiRoC was trained entirely in simulation on a multi-stage placement task (“pick up correction fluid and place it into box layer 2”). Deployed directly to a physical robotic arm without any real-world fine-tuning or hardware adaptation, the policy successfully executed the full multi-stage sequence under planner guidance.

---

Ablation Studies: Evaluating Module Contributions

Kong et al. conducted component ablations on the LIBERO-Object benchmark alongside cross-validation evaluations of the high-level planner.

Component Ablation Findings (LIBERO-Object)

• Full HiRoC Framework: Achieves a 96.0% success rate.
• Without Local GRPO ($w/oL$): Drops to 95.20%, confirming that subgoal-level advantage signals refine motor convergence.
• Without High-Level Planner ($w/oP$): Drops to 92.60%. The learning curves exhibit severe instability and loss fluctuations when forced to rely on flat global instructions.
• Without Global GRPO ($w/oG$): Collapses to 4.0%, proving that trajectory-level task completion rewards are mandatory to anchor online reinforcement learning.

Planner Accuracy Cross-Validation

The fine-tuned Qwen2.5-VL-3B planner was cross-validated against general multi-step planning baselines (RoboBrain2-3B and RoboBrain2-7B) across varying trajectory lengths:

Trajectory Category	HiRoC Planner Subgoal Accuracy	RoboBrain2-3B Subgoal Accuracy	RoboBrain2-7B Subgoal Accuracy
Short Trajectories	94.92%	0.04%	0.29%
Medium Trajectories	95.20%	0.06%	0.70%
Long Trajectories	95.09%	0.06%	0.98%
Mixed Trajectories (Average)	95.07%	0.06%	0.66%

These empirical evaluations show that general multi-step foundation models fail at fine-grained robotic subgoal decomposition (<1% accuracy) without target domain supervised post-training ($D_{plan}$).

---

Key Takeaways for Technical Practitioners

1. Decoupled Hierarchies Mitigate Error Accumulation: Separating high-level visual language planning from low-level motor execution prevents compounding trajectory errors in multi-stage manipulation workflows by supplying temporal sub-goal anchors.
2. Pre-RL Distribution Alignment Prevents Execution Paralysis: Supervised fine-tuning on intermediate state-subgoal-action pairs ($D_{exe}$) is essential to align the executor with planner outputs, eliminating cold-start execution paralysis before online RL tuning.
3. Convex Dual-Advantage Weighting Enhances RL Stability: Combining trajectory-level task rewards ($A^{task}_i$) with localized subgoal progress ($A^{sub}_i$) via convex adaptive weights ($w_t = \frac{L}{K+L}, w_s = \frac{K}{K+L}$) yields stable policy updates without objective drift.
4. Freezing High-Level Planners Eliminates Policy Shift: Freezing the high-level planner during low-level executor RL fine-tuning prevents moving-target distribution shift and eliminates credit assignment ambiguity under sparse reward environments.

Read the full paper on arXiv · PDF