RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation

Can Your AI Judge a Robot’s Progress?

In the current landscape of robotic manipulation, we are facing a paradoxical bottleneck. While modern Vision-Language Models (VLMs) like GPT-4o and Claude 3.5 Sonnet exhibit “God-like” perception in static image reasoning, our robots remain remarkably “blind” to the temporal and physical nuances of their own actions. For years, the industry has relied on a “Success-Only” paradigm: a binary judgment of whether a task was completed. But as we move toward autonomous systems that must self-correct, this binary view is no longer enough.

To act as effective “judges”—functioning as visual critics, online reward generators, and failure detectors—VLMs must move beyond object-centric recognition. They need to understand the latent process state: the subtle evolution of a sliding contact, the precise moment a primitive action transitions, or the temporal sequence of a complex assembly. The core problem is that existing benchmarks treat these “physically grounded execution traces” as auxiliary data. Without a diagnostic framework for process-aware understanding, we cannot bridge the gap between a robot that merely acts and one that truly understands its progress.

Introducing RoboProcessBench: A Diagnostic Vision for Manipulation

RoboProcessBench shifts the evaluative lens from action generation to process monitoring. Its mission is to diagnose whether a VLM can infer the unfolding of a manipulation execution based on visual observations and optional task context. By decomposing manipulation into 12 diagnostic question families, the benchmark provides a granular “Understanding Profile” across two primary dimensions: Static Monitoring (identifying the current state) and Dynamic Reasoning (predicting evolution and continuation).

Category	ID	Question Family	Description
Static Monitoring	T1	Phase Recognition	Identifying coarse phases (e.g., Hold & Carry).
	T2	Contact Detection	Determining if task-relevant contact is established.
	T4	Bi-arm Coordination	Identifying coordination patterns between arms.
	T10*	Current Primitive	Recognizing the active low-level primitive.
Dynamic Reasoning	T3	Motion Direction	Predicting the dominant direction of end-effector motion.
	T5	Local Progress	Judging if a step is at an early, middle, or late stage.
	T6	Motion State	Identifying if the manipulator is moving or stationary.
	T7	Outcome Prediction	Inferring if an ongoing attempt will succeed.
	T8	Temporal Ordering	Determining the chronological order of shuffled frames.
	T9	Temporal Priority	Deciding which of two frames occurred earlier.
	T11*	Next Primitive	Inferring the subsequent low-level primitive.
	T12*	Primitive Restoration	Filling a missing primitive in an action chain.

*Denotes Primitive-aware extensions specifically grounded in REASSEMBLE annotations, requiring specific low-level action supervision.

The benchmarking pipeline begins with raw ego-centric video and synchronized physical signals. These are aligned and segmented into units that map physical reality (like force/torque spikes) to visual frames, ensuring that “ground truth” process labels are rooted in physical evidence rather than subjective human interpretation.

The Data Substrate: 58k Points of Physical Grounding

The foundation of this research is the ProcessData corpus, a massive collection of ~58,000 QA pairs across 260 tasks. The “secret sauce” of this dataset lies in its use of Local Process Units (LPUs). Rather than treating a long episode as a single, opaque block, LPUs decompose the execution into auditable segments anchored by contact onset/loss or primitive transitions. This alignment allows the benchmark to provide high-leverage supervision for local state and action-chain reasoning.

To ensure breadth and diversity, ProcessData aggregates four distinct robotic sources, each playing a strategic role:

• GM-100: Supplies the “what” via long-tail, goal-conditioned manipulation tasks.
• RH20T: Provides the “how” through contact-rich, multimodal execution traces (force/torque).
• REASSEMBLE: Offers explicit, primitive-level action chains for fine-grained auditing.
• AIST-Bimanual: Focuses on the complexities of bimanual motion and coordination.

The semantic breadth of ProcessData is vast, covering a “word cloud” of manipulation verbs ranging from common actions like “pick,” “place,” and “insert” to long-tail tasks like “squeeze,” “wipe,” “peel,” and “fold.” The dataset composition reflects real-world complexity: while Pick/Place/Remove tasks comprise 33.1% of the data, the benchmark also includes significant “long-tail” categories such as Deform/Clean (9.6%), Fluid/Granular Transfer (5.0%), and Impact-based tasks (4.2%).

VLM Benchmarking: The Temporal Reasoning Bottleneck

When we subject current VLMs to RoboProcessBench, the results provide a sobering reality check. Even frontier models—including GPT-4o, Claude-3.5-Sonnet, and Gemini-3.1-Flash—exhibit a “jagged” performance profile on the diagnostic radar charts. While most models perform reasonably well at Phase Recognition (T1) and Contact Detection (T2), their performance “craters” when faced with temporal reasoning.

Key findings include:

• The Temporal Void: Models struggle immensely with temporal ordering (T8) and priority (T9), suggesting they view manipulation as a series of disconnected states rather than a continuous physical evolution.
• Primitive-Local Blindness: Estimating “local progress” (T5)—judging if a specific primitive is 20% or 80% complete—remains a major bottleneck, even for high-tier closed-source models.
• The Data Vacuum: The failure of frontier models suggests that general-purpose web-scale training is insufficient for physical process understanding.

However, the research offers a path forward through learnable supervision. By fine-tuning model backbones like Qwen2.5-VL-7B and InternVL-3-8B on the ProcessData-SFT split, researchers achieved consistent gains in motion state, progress estimation, and primitive-aware cues. This confirms that the problem isn’t model architecture; it’s the lack of structured, process-aware training signals in current datasets.

Closing the Loop: Why This Matters for the Future of Robotics

RoboProcessBench arrives at a pivotal moment. As the field moves toward Process Reward Models (PRMs)—a trend already exploding in LLM reasoning (such as OpenAI’s o1)—there is a desperate need for a “PRM-as-a-Judge” for the physical world. By providing a structured taxonomy and a physically grounded dataset, this benchmark allows us to develop “RoboPulse” style evaluators that can audit a robot’s performance step-by-step.

This shift moves us closer to robots that can detect their own failures in real-time, explain why a contact was lost, and adjust their strategy before a minor error becomes a terminal failure.

Key Takeaways:

1. Shift in Evaluation: We must move from binary “success” metrics to process-aware judging to build truly autonomous robotic critics.
2. The LPU Advantage: Anchoring Local Process Units to physical signals (force/contact) is the only way to generate objective, fine-grained understanding of manipulation.
3. The Supervision Gap: Process understanding is a learnable skill. Structured data like ProcessData-SFT can bridge the performance gap that currently plagues even the most advanced foundation models.

Reader Takeaways & Resources

• Standardized Diagnostics: Use RoboProcessBench to identify specifically where your model’s physical reasoning fails (e.g., temporal vs. static).
• SFT is Required: Natively, VLMs are poor at progress-based reasoning; specialized fine-tuning on grounded traces is the current prerequisite for “judge-like” roles.
• Future Foundation: Robust process understanding is the mandatory precursor for any VLM intended to serve as an online reward generator or failure detector.

Explore the dataset, evaluation results, and project details at: https://processbench-2026.github.io

Read the full paper on arXiv · PDF