RACF: A Resilient Autonomous Car Framework with Object Distance Correction

1. Introduction: The Fragility of Vision in Motion

In the rush toward full autonomy, we have largely ignored a terrifying reality: our most sophisticated vision systems are brittle. Obstacle distance estimation is the bedrock of safe closed-loop operation; every braking maneuver and collision-avoidance trajectory depends on it. Yet, vision-based depth sensing—our primary tool—is notoriously vulnerable to environmental degradation like sun glare and dense fog, as well as calculated adversarial attacks like physical patches designed to “hide” stop signs.

While the industry pursues rapid deployment, these “failure-first” safety challenges remain unresolved. We cannot simply rely on better detectors. The Resilient Autonomous Car Framework (RACF) represents a paradigm shift—a proactive, system-level architecture that doesn’t just recognize a failure but actively repairs the perception signal in real time.

2. The Problem: Why Standard Multi-Sensor Fusion Isn’t Enough

Multi-sensor redundancy is often mistaken for resilience. However, a “Research Gap” exists because redundancy alone does not guarantee stability under stress. Current defensive strategies typically fall into three traps:

• Image/Model-Level Defenses: Techniques like detector hardening or adversarial patch mitigation improve visual recognition but lack explicit stability for the control-critical braking signals (distance) required for safety.
• Naive/Unconditional Fusion: Traditional fusion pipelines often assume reliable sensing. When one modality—such as a depth camera—is compromised, a naive system may still incorporate that corrupted data, injecting errors directly into the vehicle’s decision policy.
• Forecast-Only Models: While time-series models provide temporal priors, they lack a “repair-only” mechanism. They function as black-box predictors rather than conservative safeguards that preserve original, clean data.

3. The RACF Architecture: Redundancy Meets Diversity

The RACF architecture addresses these gaps by introducing a Trust & Safety Layer. This layer acts as a gatekeeper, evaluating the consistency of incoming signals before they reach downstream control functions. Resilience is achieved through a combination of high-resolution redundancy and physical diversity.

Source of Evidence	Method Category	Primary Role
Depth Camera	Learned Sensor	High-resolution distance data (Primary Redundancy).
LiDAR	Geometric Range	2D sparse returns used as a consistency trigger (Redundancy & Grounding).
Physics/Kinematic Model	Physics-Aware Constraints	Vehicle dynamics ($\Delta d \approx -v\Delta t$) ensure temporal plausibility (Diversity).

The key innovation here is using the 2D LiDAR as a “grounding” mechanism. Because 2D LiDAR often suffers from sparse returns and limited angular resolution on platforms like the QCar 2, we do not use it as a primary distance source. Instead, it serves as a physical reference to validate the vision-based depth estimate.

4. Deep Dive: The Object Distance Correction Algorithm (ODCA)

The core of our perception-layer defense is the Object Distance Correction Algorithm (ODCA). It is designed around the principle of Nominal Preservation: if sensors agree, the system does nothing.

The Cross-Sensor Gate and Convex Fusion

The framework monitors the residual $r_{XS}(t)$, defined as the absolute difference between the depth estimate $\tilde{d}(t)$ and the aligned LiDAR range $l_{\to D}(t)$. The gate weight $w(t)$ is computed using two thresholds, $\tau_{low}$ and $\tau_{high}$:

$w(t) = \text{clip}\left(\frac{r_{XS}(t) - \tau_{low}}{\tau_{high} - \tau_{low}}, 0, 1\right)^\gamma$

As per Proposition 1, if $r_{XS}(t) \le \tau_{low}$, then $w(t) = 0$, and the original measurement is passed through unchanged ($d_{fused}(t) = \tilde{d}(t)$). If disagreement grows, the system performs a convex combination to produce the final distance:

$d_{fused}(t) = (1 - w(t)) \tilde{d}(t) + w(t) \hat{d}_{rep}(t)$

ChronosV2 and the Delta Repair Head

To generate the repair signal $\hat{d}_{rep}(t)$, we utilize ChronosV2, a frozen foundation time-series model. By keeping the model frozen, we capture universal temporal priors without the overhead of task-specific retraining. We then append a lightweight Delta Repair Head—comprising only 921 parameters—to predict an additive correction $\Delta(t)$. This “Delta” formulation ensures that under nominal conditions, the correction is driven toward zero.

Physics-Aware Training

During training, we apply a kinematics regularizer ($L_{kin}$) to the repaired signal. This ensures the output is physically plausible, adhering to the constraint that change in distance should roughly equal negative velocity multiplied by the change in time ($\Delta d \approx -v\Delta t$).

5. Empirical Results: Proving Resilience on the QCar 2 Platform

We validated RACF using the Quanser QCar 2 platform, utilizing YOLOv8-seg for stop-sign localization and DBSCAN for LiDAR clustering.

1. RMSE Reduction: Under strong corruption regimes (bias, confidence degradation, and blackouts), RACF achieved up to a 35% reduction in RMSE compared to vanilla depth sensing.
2. Resilience Metrics: We evaluated the framework using Bounded Degradation (BD) and the Resilience Gain Ratio (RGR). RACF demonstrated significantly more stable performance across weak, mid, and strong attack strengths compared to forecasting baselines like SOFTS, NHITS, and DLinear.
3. Real-Time Efficiency: Despite the foundation model backbone, the framework maintained an average processing cycle of approx. 49ms (20 Hz). This meets the real-time constraints required for high-speed autonomous maneuvers.

6. Real-World Impact: Stop-Sign Compliance and Safety Metrics

To test the “Trust & Safety Layer” in the wild, we conducted closed-loop trials where physical adversarial patches were used to suppress stop-sign detection during critical braking windows.

Metric	Without Defense (Attack)	With RACF (Attack + ODCA)
Stop Compliance Rate (SCR)	0.17	0.80
Braking Latency	2.65 seconds	0.19 seconds

The results are stark. Without RACF, the vehicle suffered from an Attack Success Rate (ASR) of 0.83, frequently failing to brake or braking far too late. With RACF, braking latency was slashed by over 90%.

It is important to note the Reaction-Time Bound: the 20% failure rate in the “With RACF” setting occurred primarily during sustained 3-second attacks. In these cases, the attack persistence exceeded the system’s causal recovery horizon, highlighting that while RACF is remarkably resilient, absolute safety requires a multi-layered defense strategy.

7. Conclusion: A Blueprint for Safety-Critical Autonomy

The Resilient Autonomous Car Framework (RACF) proves that we can build robust AI for safety-critical systems without massive, opaque retraining cycles. By combining the universal temporal priors of frozen foundation models with conservative, consistency-triggered repair, we create a system that respects the physics of the real world.

This principle of conservative sensor substitution is not limited to ground vehicles; it serves as a transferable blueprint for any autonomous system, from UAV collision avoidance to industrial robotics. In the future of AI safety, the most critical component won’t just be how well a system sees, but how well it knows when its own eyes are failing.

Read the full paper on arXiv · PDF