Shift & Drift: A Zero-Shot Benchmark for Generalizable and Robust Autonomous Driving Motion Planning

1. Introduction: The Mirage of Benchmark Performance

Current advancements in learning-based autonomous driving (AD) planners are predominantly measured against standardized benchmarks such as nuPlan and the Waymo Open Dataset. While these environments have catalyzed progress, research by Canevaro et al. suggests that high scores in these frameworks often represent an “optimistic upper bound” of performance rather than true deployment readiness.

The researchers identify a critical gap between high performance in familiar training environments and reliability in the real world. This discrepancy is frequently driven by geographic overfitting—where a model implicitly memorizes specific map topologies—and “causal confusion,” where planners latch onto spurious correlations in training data instead of transferable driving priors. These factors mask planning fragilities that only emerge when an agent is forced to navigate outside its training distribution. To address these systemic blind spots, Canevaro et al. introduced the “Shift & Drift” benchmark, a dual-track framework designed to stress-test motion planning generalization and robustness through zero-shot evaluation.

2. Track 1: The Semantic Shift and the DeepPlan Pipeline

The first axis of the benchmark, the “Semantic Shift Track,” evaluates how planners handle novel environments and agent behaviors never encountered during training. To facilitate this, Canevaro et al. developed “DeepPlan,” a suite of scenarios created by converting the aerial DeepScenario Open 3D (DSC3D) dataset into the nuPlan framework. This enables a rigorous zero-shot evaluation where models trained on North American and Singaporean data are tested against unfamiliar German and San Franciscan environments.

A critical technical contribution of this track is the conversion pipeline, which maps OpenDRIVE files to a GeoPackage format required for nuPlan simulation. Unlike standard datasets derived from ego-centric sensor logs, which are prone to occlusions, the DSC3D aerial data provides occlusion-free trajectories. This addresses the “phantom empty space” failure mode: in standard simulators, when a planner deviates from a recorded path, it may enter areas that were blind spots during the original recording. This causes the agent to navigate through “phantom” space where actors were physically present but not captured, leading to an overestimation of safety.

Canevaro et al. highlight three core challenges within this track:

• Novel Urban Topologies: Planners face road layouts and intersection types not present in training data across 1,182 scenarios spanning San Francisco and four German cities: Munich, Stuttgart, Sindelfingen, and Berlin.
• Dense Social Interactions: Scenarios feature significantly higher densities of pedestrians and cyclists than standard logs, testing multi-agent reasoning.
• Regional Driving Styles: The evaluation exposes models to localized European driving norms that diverge from North American training distributions.

3. Track 2: The State-Distribution Drift and Compounding Errors

The second track addresses “State-Distribution Drift,” targeting the “compounding error problem.” This phenomenon occurs when small, cumulative inaccuracies in execution push an agent into off-policy states from which it lacks the recovery behaviors to return. To quantify this, Canevaro et al. inject stochastic perturbations into the ego-vehicle’s dynamics, forcing the planner to utilize receding-horizon feedback to actively compensate for accumulated error.

The researchers utilize two distinct noise models to simulate real-world execution imperfections:

Noise Mechanism	Description	Real-World Analog
Additive White Gaussian Noise (AWGN)	High-frequency, temporally uncorrelated jitter added to command outputs.	Mechanical jitter and instantaneous control inaccuracies.
Ornstein-Uhlenbeck (OU) Process	Low-frequency, temporally correlated noise with a mean-reversion rate of $\theta = 2.0 s^{-1}$ and a 0.5 s correlation time.	Systematic errors such as wheel misalignment, crosswinds, or environmental forces.

According to Canevaro et al., surviving the OU process requires a level of dynamic stability often ignored in standard, noise-free simulations, as errors persist and accumulate across multiple frames.

4. The Showdown: Comparing Motion Planning Paradigms

Canevaro et al. conducted a systematic evaluation of diverse planning paradigms, revealing fundamental differences in how they handle distribution shifts. Performance is measured primarily via the Non-Collision Rate (NCR) and Progress Along Expert Route (PER).

• Imitation Learning (IL) (e.g., PlanTF, Diffusion Planner): These models demonstrated a “severe sensitivity” to semantic shift. Driven by causal confusion—where models latch onto spurious training correlations rather than driving logic—Diffusion Planner’s performance dropped by 76% in the pedestrian-dense environments of Munich. Under OU noise, IL models suffered from “persistent drift” failures, as they lack the closed-loop recovery mechanisms necessary to correct accumulated multi-frame errors.
• Rule-Based Planners (e.g., PDM-Closed): These served as a middle ground. While they achieved the highest safety scores (NCR), they exhibited low progress metrics (PER). Canevaro et al. attribute this to a failure of “adaptive social reasoning”; the rigid heuristics prioritize safety margins to such a degree that they cannot fluidly negotiate the dense interactions found in European urban centers.
• Reinforcement Learning (RL) (e.g., CaRL): This paradigm showed the most resilience, maintaining high scores across both tracks with a performance decay of at most 8% even under high-intensity drift. This resilience likely stems from the stochastic nature of RL training, which forces agents to visit and recover from off-policy states during the learning process, effectively functioning as a dynamics-aware governor.

5. Critical Insights: The Trade-off Between Fidelity and Resilience

The findings of Canevaro et al. culminate in three critical takeaways regarding the current state of autonomous motion planning:

1. The Generalization Gap: High in-distribution performance in IL models does not guarantee safety in novel urban topologies or dense social settings. Models often memorize maps rather than learning transferable driving priors.
2. The Robustness Paradox: Standard evaluations mask control instabilities. Planners may appear safe in noise-free simulations but fail catastrophically when subjected to minor, correlated actuation noise that simulates hardware imperfections.
3. Heuristic Rigidity vs. Learned Adaptability: While rule-based systems provide safety guarantees, their manual heuristics are too inflexible for complex social navigation. Conversely, RL-based agents demonstrate superior learned adaptability and recovery.

Canevaro et al. explicitly identify an “empirical trade-off between imitation fidelity and closed-loop resilience.” This Robustness Paradox suggests that models optimized solely to mimic expert trajectories often sacrifice the robust recovery behaviors essential for real-world deployment.

6. Conclusion: A New Standard for Deployment Readiness

The “Shift & Drift” benchmark provides a rigorous framework for assessing an agent’s social intelligence and dynamic stability. By demonstrating that standard benchmarks act as an “optimistic upper bound,” Canevaro et al. highlight that true reliability requires the ability to handle both semantic novelty and physical uncertainty. High scores on standard benchmarks are often a product of geographic overfitting rather than actual competence.

Looking ahead, Canevaro et al. envision scaling this benchmark to include a broader variety of global driving cultures. Future research will focus on investigating hybrid architectures that seek to combine the safety-critical guarantees of rule-based systems with the resilient exploration and adaptability found in reinforcement learning.

Read the full paper on arXiv · PDF