MAMMOTH: A Multi-Modal End-to-End Policy for Off-Road Mobility Robust to Missing Modality

1. The Perception Crisis in Off-Road Environments

Autonomous off-road navigation is currently hindered by extreme terrain diversity, drastic illumination changes, and the persistent threat of sensor degradation. Kotian et al. identify a significant “perception crisis” in the field, noting that while recent learning-based methods have made strides in structured settings, they struggle in the unstructured “wilds” of forests, caves, and nighttime environments.

A primary failure point is the heavy reliance on RGB-heavy policies, such as ViNT or NoMaD. The authors observe that these systems often fail when encountering “visually degraded conditions”—scenarios featuring intense glares, deep shadows, or total darkness. Furthermore, existing multi-modal systems typically suffer from a “rigid dependency” on perfect sensor inputs. According to the paper, if a single sensory channel fails or provides noisy data due to environmental stress, the entire policy often collapses, leaving the robot unable to navigate safely.

2. Introducing MAMMOTH: A Blueprint for Resilience

To address these vulnerabilities, the authors developed MAMMOTH (MAsking Multi-Modal inputs for Off-road Traversability Heuristic-informed navigation). This unified end-to-end navigation policy is designed for both visual-goal-conditioned navigation and undirected exploration.

As detailed in the methodology section, MAMMOTH is built upon three core design principles:

• Synergistic Multi-Modal Fusion: The system integrates inputs from a diverse sensor suite to leverage complementary scene information, using a specialized backbone to weight modalities based on environmental context.
• Flexible and Robust Deployment: By utilizing a training strategy that anticipates sensor failure, the model maintains operational integrity even when specific inputs are missing or degraded.
• Physically-Aware Trajectory Generation: Rather than relying solely on geometric pathfinding, the policy incorporates a heuristic that accounts for the physical consequences of terrain traversal, such as vehicle stability.

3. The Architecture of Adaptability: MoE and Modality Dropout

MAMMOTH utilizes a comprehensive suite of sensors, including RGB imagery, thermal Long-Wave Infrared (LWIR), 3D LiDAR pointclouds, and ego velocity. To process this data, the authors employ a sparse Mixture-of-Experts (MoE) backbone featuring Laplace gating. This “Synergistic Fusion” mechanism utilizes four lightweight routers—one for each modality—to dynamically select a subset of experts from a shared pool. This allows the model to weight different modalities based on the environment, such as prioritizing thermal and LiDAR data in low-light conditions where RGB data is unreliable.

To ensure system-level robustness, the authors implemented a “Modality Dropout” training strategy. This “mask-and-ignore” approach involves randomly dropping specific sensor tokens during training. This forces the model to learn cross-modal representations, ensuring it does not over-rely on any single modality and remains functional if a sensor fails during real-world deployment.

Input Modality	Encoder Architecture	Data Type Processed
RGB Images	EfficientNet-B0	96x96x3 visual frames
Thermal LWIR	EfficientNet-B0	96x96x1 thermal frames
3D LiDAR	PointPillars	Distilled into a 256-dimensional vector representation
Ego Velocity	Fourier Features + MLP	Velocity norm vector

4. Beyond Geometry: The Role of the Traversability Heuristic

The authors argue that simple geometric planning is insufficient for off-road environments. To achieve terrain-aware navigation, MAMMOTH incorporates an “Intrinsic Traversability Heuristic” derived from the Power Spectral Density (PSD) of vertical (Z-axis) acceleration measured by an onboard IMU.

This heuristic is integrated into a conditional Denoising Diffusion Probabilistic Model (DDPM), which generates candidate trajectories. The noise prediction network within this architecture is a 1-D Conditional U-Net. The policy evaluates these trajectories not just for their path to a goal, but for their physical safety. The trajectory selection process follows this ranking method:

1. The diffusion policy generates $N$ candidate trajectories based on sensory and goal context.
2. The model estimates the associated physical cost (traversability) for each path.
3. The system ranks the candidates and selects the “smoother, safer” path that minimizes the risk to vehicle stability.

5. Empirical Evidence: Stress-Testing in the Forest

The authors validated MAMMOTH using the “Copernicus” all-terrain robot in cluttered, forest-like environments characterized by dense overhead trees and a lack of clear trails. The system was stress-tested in both daytime and nighttime conditions and compared against established baselines like NoMaD and FlowNav.

As reported in the paper’s experimental results, MAMMOTH demonstrated superior performance, particularly in avoiding collisions and maintaining stability on complex terrain.

Performance Comparison: MAMMOTH vs. NoMaD Baseline

Metric	Condition	NoMaD (Baseline)	MAMMOTH (Proposed)
Collision Rate (CR) ↓	Day	1.3	0.7
Collision Rate (CR) ↓	Night	Not Operational	0.8
Undesirable Behaviors (UB) ↓	Day	1.0	0.6
Undesirable Behaviors (UB) ↓	Night	Not Operational	0.75
Success Rate (SR) ↑	Day	17%	33%

Note: CR represents collisions per minute; UB represents instances of unsafe terrain choice or manual takeover.

6. Fail-Operational Performance: Handling Missing Sensors

A critical component of the research involved “Missing Modality Ablations.” By deactivating individual sensors during autonomous runs, Kotian et al. quantified how the system adapts to hardware failure.

The findings from Tables II and III indicate that different modalities serve specialized roles:

• 3D LiDAR: Found to be crucial for collision avoidance; deactivating this sensor led to a sharp increase in the Collision Rate (CR).
• Thermal and Ego Velocity: Essential for terrain understanding; these sensors significantly reduced Undesirable Behaviors (UB), especially in nighttime scenarios where RGB data provided no utility.

The authors conclude that the “mask-and-ignore” training allows for zero-shot generalization, meaning the robot can be deployed on platforms missing specific sensors (like LiDAR or thermal cameras) while still maintaining a baseline of safe operation.

7. Conclusion and Research Takeaways

Kotian et al. demonstrate that MAMMOTH provides a robust framework for off-road navigation by moving away from a rigid reliance on any single sensor. Their results suggest that combining multi-modal fusion with generative diffusion models and physical heuristics creates a more resilient embodied AI system.

For AI safety researchers and field robotics practitioners, the paper offers three primary takeaways:

1. Redundancy via Dropout: Training models to explicitly ignore missing modalities prevents the total policy failure commonly seen in rigid multi-modal systems.
2. Synergistic vs. Individual Utility: Multi-modal systems should be designed to weigh inputs dynamically (e.g., via the per-modality router MoE) to adapt to changing environmental conditions like sudden darkness or glare.
3. Physical Heuristics as Safety Constraints: Integrating self-supervised physical costs (like IMU-derived traversability) into trajectory generation improves safety without the high computational overhead of post-hoc guidance.

The authors have committed to making their code and the associated off-road dataset publicly available to support further research in robust autonomous mobility.

Read the full paper on arXiv · PDF