RedFlow: Redirect Failure into Action-level Corrections for Flow-matching VLA Policy

1. Introduction: The Evaluative Trap in Embodied Post-Training

Post-training Vision-Language-Action (VLA) policies through reinforcement learning (RL) enables robotic systems to improve beyond supervised imitation by leveraging deployment experience. However, standard reward- and preference-based RL methods fall into an “evaluative trap.” These approaches evaluate or rank executed trajectories—flagging desirable versus undesirable outcomes—without providing explicit, localized corrective signals that specify how a failed action chunk should be modified. Consequently, the policy learns which behaviors to avoid, but receives no explicit directive on what alternative action to execute instead. This leads to underutilized failure trajectories and severely limits sample efficiency during fine-tuning.

To provide direct action-level corrections, alternative paradigms employ Human-in-the-Loop (HIL) intervention frameworks. In HIL systems, human operators manually correct the robot when it deviates from a safe or successful trajectory. While HIL provides high-quality corrective targets, it is labor-intensive, expensive, and difficult to scale across varied robotic tasks, environments, and hardware platforms.

To address this challenge without online human intervention or additional data collection, Yan et al. propose RedFlow, an offline post-training framework designed specifically for flow-matching VLA policies. RedFlow extracts localized corrective action targets directly from fixed, mixed-quality rollout buffers. By matching execution contexts across trajectories with different outcomes, Yan et al. demonstrate that higher-quality action chunks executed in similar states can serve as corrective targets for failing actions without requiring additional environment interactions or human interventions.

Central Thesis: Standard preference and reward-based post-training methods leave failure rollouts underutilized by providing purely evaluative feedback. RedFlow bridges this gap by matching execution contexts across fixed, mixed-quality offline rollouts—converting higher-quality action chunks executed at similar task stages and robot configurations into explicit, localized supervisory references for failing actions.

---

2. Overcoming State Disparity: Execution-Context Matching

A key conceptual foundation for reusing failure trajectories is trajectory stitching: connecting a failed trajectory to a successful one at a shared state. However, Yan et al. observe that traditional trajectory stitching fails in complex robotic manipulation because exact state matches across independent rollouts are virtually non-existent. Minor variations in object placement, gripper approach vectors, contact dynamics, or visual occlusions ensure that two robot rollouts rarely share an identical environment state. Furthermore, stitching an entire remaining trajectory replaces the whole future action sequence without pinpointing the exact failure point or identifying the minimal localized action correction needed.

To overcome the lack of exact state matches, Yan et al. propose Execution-Context Matching. Instead of requiring full image-level or state-level identity, this module constructs a compact, low-dimensional execution-context vector $f_t$ for each action chunk:

$f_t = [\tilde{q}_t; \beta \bar{p}_t]$

where:

• $\tilde{q}_t$ represents the robot’s normalized proprioceptive state (joint or end-effector configuration), acting as a spatial configuration key.
• $\bar{p}_t$ represents the smoothed task progress estimate, acting as a high-level temporal stage aligner across different rollouts.
• $\beta > 0$ is a scaling hyperparameter balancing the relative magnitudes of proprioception and task progress.

Combining these two complementary features enables the framework to group action chunks executed at similar functional stages and robot poses without requiring identical visual pixels, object poses, or contact dynamics.

To compute task progress zero-shot without task-specific fine-tuning, the authors utilize the pretrained Robo-Dopamine 2.0 General Reward Model (GRM). To eliminate high-frequency fluctuations in raw GRM predictions across consecutive frames, task progress is smoothed using a box filter of half-window size $W$:

$\bar{p}_t = \frac{1}{|J_t|} \sum_{j \in J_t} R(o_j, l, H_j), \quad J_t = \{j : \max(0, t-W) \le j \le \min(T-1, t+W)\}$

where $R(o_j, l, H_j)$ is the GRM progress evaluation given historical visual frame sequence $H_j$, language instruction $l$, and observation $o_j$.

Yan et al. then apply the HDBSCAN density-based clustering algorithm to cluster these compact context vectors $\{f_t\}$ across all action chunks in the mixed-quality offline buffer $\mathcal{D}$. This groups action chunks executed at comparable task stages and robot configurations into context clusters $\{C_c\}$. Unassigned action chunks that fail density thresholds are designated as outliers.

Property / Dimension	Traditional Trajectory Stitching	RedFlow Execution-Context Matching
Match Assumption / Key	Assumes exact state identity across trajectories ($s_t = s_t'$)	Uses compact context key $f_t = [\tilde{q}_t; \beta \bar{p}_t]$ combining progress and proprioception
Input Domain & Space	High-dimensional raw observation space (images + full physical states)	Compact low-dimensional context space balancing temporal stage and robot pose
Correction Granularity & Scope	Trajectory-level: replaces the entire remaining action sequence	Chunk-level: provides localized action redirection toward cluster-weighted centroids $a^*_t$
Handling of Outliers / Unmatched States	Fails to match; leaves unmatched states uncorrected	HDBSCAN explicitly designates sparse chunks as outliers, falling back to failure suppression
Sensitivity to Visual & Contact Variations	Extremely high; minor visual or contact shifts break state matching	Low; invariant to image pixel variations, fine object shifts, and contact dynamics

---

3. The Mathematics of Action Redirection: Quality Scores and Loss Formulation

Flow-Matching Preliminaries & Endpoint Reconstruction

A flow-matching VLA policy $\pi_\theta(a_t | o_t, l)$ generates an action chunk $a_t = x_0 \in \mathbb{R}^{K \times D}$ (comprising $K$ consecutive $D$-DoF control commands) by transporting a Gaussian noise sample $x_1 \sim \mathcal{N}(0, I)$ to the clean data endpoint $x_0$ along a probability path defined over flow time $n \in [0, 1]$. Adopting a linear probability path:

$x_n = (1 - n)x_0 + n x_1, \quad x_1 \sim \mathcal{N}(0, I), \quad n \sim U(0, 1)$

the target velocity field is given by $u_n = x_1 - x_0$. The policy network parameterizes a conditional velocity field $v_\theta(x_n, n, o_t, l)$ trained to approximate $u_n$.

Given a noisy action sample $x_n$ at flow time $n$, the policy’s predicted clean action endpoint $\hat{x}_0$ is reconstructed analytically:

$\hat{x}_0 = x_n - n v_\theta(x_n, n, o_t, l)$

Geometric Intuition of Endpoint Redirection

Rather than applying corrective constraints or velocity perturbations directly to $v_\theta$ or the noisy path variable $x_n$, RedFlow operates explicitly on the reconstructed endpoint $\hat{x}_0$. Because $\hat{x}_0$ lies directly in the physical action space $\mathcal{A}$, geometric concepts—such as safety distance margins ($m$) around failed actions and quality-weighted corrective target centroids ($a^*_t$)—remain geometrically invariant to flow time $n$. Operating in $\mathcal{A}$ ensures that supervisory signals represent physical action targets regardless of the internal flow time step.

Signed Chunk-Quality Score ($\hat{A}_t$)

To evaluate the quality of individual action chunks within offline rollouts, Yan et al. calculate a retrospective, signed chunk-quality score:

$\hat{A}_t = \bar{p}_{\min(t+W, T-1)} - \bar{p}_{\max(t-W, 0)} + b \cdot (2 \cdot \mathbf{1}[y_\tau = 1] - 1)$

where $b > 0$ weights the global episode outcome $y_\tau \in \{0, 1\}$, and the progress-difference term captures localized task progress changes. Chunks with $\hat{A}_t > 0$ are classified as positive ($C_c^+$), while chunks with $\hat{A}_t < 0$ are classified as negative. Yan et al. emphasize that $\hat{A}_t$ is a retrospective quality indicator for relative weighting rather than a causal, counterfactual advantage function.

Corrective Target Construction & Correctability Indicator ($c_t$)

For a negative action chunk $a_t$ residing in context cluster $C_c$, if the cluster contains positive chunks ($C_c^+ \neq \emptyset$), the authors construct a corrective target $a^*_t$ as a quality-weighted centroid of those positive actions:

$\alpha_i = \frac{\exp(\hat{A}_i / \kappa)}{\sum_{j \in C_c^+} \exp(\hat{A}_j / \kappa)}, \quad a^*_t = \sum_{i \in C_c^+} \alpha_i a_i$

where $\kappa > 0$ controls the soft-max temperature emphasizing higher-quality actions.

The correctability of a negative chunk is formally defined by the logical condition:

$c_t = \mathbf{1}[\hat{A}_t < 0 \land C_c^+ \neq \emptyset]$

If a negative chunk is designated as an HDBSCAN outlier or belongs to a context cluster lacking positive support ($C_c^+ = \emptyset$), it receives $c_t = 0$. Such chunks are treated as uncorrectable and receive failure suppression without target-guided correction.

Objective Function Decomposition

The total post-training objective formulated by Yan et al. modifies standard flow-matching optimization through three additive components:

$\mathcal{L} = \mathbb{E}_{(o_t, a_t, l) \sim \mathcal{D}, x_1 \sim \mathcal{N}(0, I), n \sim U(0, 1)} \left[ \mathcal{L}_{att} + \mathcal{L}_{sup} + \mathcal{L}_{cor} \right]$

• Quality-Weighted Attraction ($\mathcal{L}_{att}$): Modulates standard flow-matching imitation loss using a soft quality weight $w_t = \sigma(\hat{A}_t / T_w) \in (0, 1)$ where $T_w$ is a temperature hyperparameter: $\mathcal{L}_{att} = w_t \| v_\theta(x_n, n, o_t, l) - u_n \|_2^2$ This preserves high-quality actions while softening the imitation objective on low-quality chunks.

• Failure Suppression ($\mathcal{L}_{sup}$): Discourages the policy from repeating observed negative actions ($\hat{A}_t < 0$) by pushing predicted clean action endpoints $\hat{x}_0$ away from negative chunks $a_t$: $\mathcal{L}_{sup} = \mathbf{1}[\hat{A}_t < 0] \lambda_{sup} (1 - w_t) \max(0, m - e_t)$ where $e_t = \|\hat{x}_0 - a_t\|_2^2$ is the endpoint reconstruction error, $\lambda_{sup}$ scales suppression strength, and $m$ is an adaptive margin. To maintain numerical stability across gradient accumulation steps, margin $m$ is updated once per logical optimizer update $k$ using detached micro-batch statistics $\bar{e}_k$: $\bar{e}_k = \frac{1}{|B_k|} \sum_{t \in B_k} \text{stopgrad}(e_t), \quad m_{k+1} = (1 - \rho) m_k + \rho s_m \bar{e}_k$ where $\rho \in (0, 1]$ is a running-average coefficient and $s_m$ is a scale factor. The stopgrad operator detaches historical reconstruction errors from the computation graph, preventing backpropagation through prior margin states across updates.

• Target-Guided Correction ($\mathcal{L}_{cor}$): Pulls predicted clean endpoints $\hat{x}_0$ toward the matched corrective target centroid $a^*_t$ for correctable negative chunks ($c_t = 1$): $\mathcal{L}_{cor} = c_t \lambda_{cor} (1 - w_t) \|\hat{x}_0 - a^*_t\|_2^2$ where $\lambda_{cor}$ controls correction strength.

Theoretical Grounding

Yan et al. provide theoretical analysis for the interaction between suppression and correction terms at a fixed flow time $n$:

Theorem 1 (Bounded endpoint redirection): Consider a predicted clean action endpoint $h \in \mathcal{A}$, a negative action chunk $a^- \in \mathcal{A}$, and a corrective target $b \in \mathcal{A}$. If $b \neq a^-$, $m > 0$, and $\lambda_{sup} \ge \lambda_{cor} > 0$, define the subsystem objective: $\phi(h) = \lambda_{cor}\|h - b\|_2^2 + \lambda_{sup}[m - \|h - a^-\|_2^2]^+$ where $[r]^+ = \max(r, 0)$. The minimizer of $\phi(h)$ projects the prediction onto the point closest to $b$ while constraining it to lie outside the safety hyper-ball of radius $\sqrt{m}$ centered at $a^-$. Equivalently, it is the exact constrained projection of $b$ onto $\{h : \|h - a^-\|_2^2 \ge m\}$.

---

4. Empirical Evaluation: Simulation and Real-Robot Results

Yan et al. evaluated RedFlow across four simulation suites in LIBERO (1,536 fixed rollouts per suite) and three real-robot manipulation tasks on an Agilex Cobot Magic hardware platform.

LIBERO Benchmark Performance

Across all four 10-task LIBERO suites, RedFlow achieved superior success rates compared to established offline policy improvement baselines initialized from the same base policy ($\pi_0$).

Method	Spatial (%)	Object (%)	Goal (%)	Long (%)	Average (%)
Base Policy	63.6	61.6	48.6	50.8	56.2
AWR	71.2	66.8	57.8	53.4	62.3
DPO	66.8	64.8	51.8	51.2	58.7
CQL	70.4	63.0	52.6	52.0	59.5
IQL	71.6	63.4	55.4	52.8	60.8
RedFlow (Ours)	75.8	70.4	71.2	55.2	68.2

RedFlow achieved an overall average success rate of 68.2% across all four suites, outperforming the base policy by +12.0 percentage points and the strongest offline baseline (AWR at 62.3%) by +5.9 percentage points.

Ablation Benchmark Results

To evaluate specific module contributions, Yan et al. conducted comprehensive ablations across the 3-suite subset consisting of Spatial, Object, and Goal. Note that while the full 4-suite RedFlow average is 68.2%, its baseline score on this 3-suite ablation subset is 72.5%.

Ablation Variant	Spatial (%)	Object (%)	Goal (%)	3-Suite Avg (%)
Full RedFlow Baseline	75.8	70.4	71.2	72.5
Training Data Composition
w/o low-quality rollouts	71.4	67.4	65.2	68.0
w/o high-quality rollouts	64.4	65.8	57.0	62.4
Execution-Context Matching
w/o task progress ($\bar{p}_t$)	50.8	46.2	33.6	43.5
w/o proprioceptive state ($\tilde{q}_t$)	62.4	60.2	52.0	58.2
Quality-Guided Action Redirection
w/o correctability filtering ($c_t$)	66.4	65.8	50.8	61.0
w/o target correction ($\mathcal{L}_{cor}$)	70.4	64.4	55.6	63.5
w/o failure suppression ($\mathcal{L}_{sup}$)	70.8	66.8	68.8	68.8
w/o suppression & correction ($\mathcal{L}_{sup} + \mathcal{L}_{cor}$)	63.8	62.8	70.4	65.7

Key insights from these empirical ablations include:

• Complementary Context Keys: Removing task progress from the context vector caused the sharpest performance drop (degrading 3-suite average success to 43.5%), confirming that temporal stage alignment is critical. Removing proprioception dropped performance to 58.2%, proving both keys are necessary.
• Target Guidance vs. Pure Repulsion: Removing target correction ($\mathcal{L}_{cor}$) reduced performance to 63.5%, whereas removing both suppression and correction ($\mathcal{L}_{sup} + \mathcal{L}_{cor}$) yielded 65.7%. This isolates a +6.8 percentage point gain directly attributable to target-guided redirection beyond quality-weighted attraction alone, and demonstrates that failure suppression benefits from an explicit positive target direction rather than unguided repulsion.

Threshold-Matched Sample Efficiency

On the LIBERO-Spatial suite, RedFlow achieved a 75.8% success rate using its fixed buffer of 1,536 offline rollouts without online interaction. Yan et al. compared this sample efficiency against online RL methods (PPO, GRPO, DDPO) trained on fresh online rollouts:

• PPO: Required 13,312 fresh rollouts ($8.7\times$ additional interactions) to reach 75.8%.
• GRPO: Required 16,384 fresh rollouts ($10.7\times$ additional interactions) to reach 75.8%.
• DDPO: Required 24,576 fresh rollouts ($16.0\times$ additional interactions) to reach 75.8%.

Real-Robot Hardware Deployments

Yan et al. deployed RedFlow on an Agilex Cobot Magic dual-arm robot platform across three complex manipulation tasks, evaluating 100 trial runs per task.

Task	Base Policy (%)	DPO (%)	AWR (%)	RedFlow (Ours) (%)
Clothes Folding	36.0	48.0	41.0	67.0
Object Sweeping	63.0	66.0	69.0	73.0
Table Cleaning	71.0	78.0	76.0	84.0
Average	56.7	64.0	62.0	74.7

RedFlow improved average real-robot manipulation success from 56.7% to 74.7% (+18.0 percentage points). On the Clothes Folding task (+31.0 point gain), qualitative analysis revealed emergent recovery behavior: when the base policy repeatedly attempted an unreachable right-arm grasp, RedFlow executed a corrective maneuver—using the left arm to pull the fabric back into a reachable workspace before completing the fold.

---

5. Architectural Limitations & Key Takeaways

Core Takeaways

• Offline Supervisory Reuse: Fixed, mixed-outcome rollout buffers contain implicit localized corrections. Matching execution contexts across rollouts converts underutilized failure trajectories into explicit supervisory action targets without online environment interaction or costly human interventions.
• Target Guidance over Pure Repulsion: Explicitly pulling failed action predictions toward positive target centroids ($\mathcal{L}_{cor}$) yields significantly higher performance gains than applying negative repulsion ($\mathcal{L}_{sup}$) or quality-weighted reweighting alone.
• Action-Space Invariance: Operating loss objectives directly on reconstructed clean endpoints $\hat{x}_0$ keeps geometric safety margins and corrective targets invariant to flow time $n$.

Technical Limitations

• Context Key Scope: The low-dimensional matching key $f_t = [\tilde{q}_t; \beta \bar{p}_t]$ omits visual features, object poses, fine contact configurations, and occlusion states. Consequently, context clustering may occasionally group action chunks executed under dissimilar environmental contact dynamics.
• Retrospective Quality Assignment: The signed quality score $\hat{A}_t$ inherits global episode outcome labels $y_\tau$. As a result, necessary intermediate maneuvers or positioning actions in ultimately failed rollouts may be mislabeled with a negative quality score.

Operational Relevance for AI Safety & Policy Correction:
RedFlow establishes an offline post-training framework that converts failed deployment rollouts into localized corrective action targets for flow-matching VLA policies. By combining zero-shot task progress and proprioceptive states to cluster execution contexts, RedFlow enables target-guided action redirection that significantly improves policy performance and sample efficiency in both simulation and physical hardware deployments—offering a scalable path toward self-improving robotic systems without online exploration risks or human intervention overhead.

Read the full paper on arXiv · PDF