Toward Robust LiDAR Semantic Segmentation for Real-World Deployment: Evaluation under Coarse Labels, Adverse Conditions, and Domain Shifts

1. Introduction: The Mirage of Clean LiDAR Benchmarks

Standard evaluation protocols for 3D LiDAR semantic segmentation rely heavily on single-domain datasets such as SemanticKITTI and nuScenes. These benchmarks measure in-distribution accuracy on clean, pre-processed point clouds using fine-grained class taxonomies. While this approach has driven steady improvements in reported mean Intersection-over-Union (mIoU) scores, the authors of a comprehensive evaluation study—Samir Abou Haidar et al. from Mines Paris and CEA List—argue that standard benchmark rankings leave physical deployment readiness largely unassessed.

In production environments, autonomous perception stacks must maintain operational safety across adverse weather, mechanical sensor degradations, electronic interference, and previously unseen geographic environments. Furthermore, perception algorithms must execute inferences within strict latency thresholds on resource-constrained embedded hardware. Standard benchmark leaderboards assume clean inputs, fixed sensor geometry, and unconstrained compute, creating an evaluation gap between high benchmark placement and real-world operational reliability.

To bridge this gap, Abou Haidar et al. introduced a three-part evaluation framework designed to test models along complementary deployment dimensions:

1. Safety-aligned coarse label taxonomies evaluating functional risk.
2. Synthetic sensing corruptions mimicking atmospheric, geometric, and hardware degradations.
3. Zero-shot cross-dataset domain shifts without adaptation.

The central thesis established by the paper is that fine-grained benchmark rankings do not reliably predict deployment readiness. When evaluated against safety-focused label groupings, environmental corruptions, and sensor domain shifts, benchmark leaderboards systematically invert, exposing significant structural vulnerabilities in top-ranked architectures.

---

2. Pillar 1: Safety-Aligned Coarse Labels vs. Benchmark Rankings

The COLA Taxonomy

To evaluate semantic segmentation from an operational safety perspective, the paper adopts the COarse-LAbel (COLA) framework. Rather than evaluating models across 16 to 19 fine-grained semantic classes, the COLA taxonomy aggregates dataset-specific classes into seven functional categories relevant to autonomous navigation:

• Driveable Ground: Surface areas designated for vehicle travel.
• Other Ground: Non-driveable terrain, sidewalks, and parking areas.
• Structure: Static architectural elements, buildings, walls, and fences.
• Vehicle: Dynamic motorized transport, including cars, trucks, buses, and construction vehicles.
• Nature: Vegetation, tree trunks, and terrain.
• Living Being: Vulnerable road users, including pedestrians, bicyclists, and motorcyclists.
• Static Object: Traffic signs, poles, barriers, and traffic cones.

Functional Safety Logic

Standard fine-grained mIoU treats all misclassifications equally. However, from a functional safety standpoint, class errors carry vastly different operational risks:

• Benign Errors: Confusing a car for a truck or a bicyclist for a motorcyclist involves geometrically and operationally similar target classes. In autonomous driving control, such intraclass errors rarely cause catastrophic failures, as downstream motion planners still identify an obstacle occupying space.
• Safety-Critical Failures: Predicting a pedestrian, cyclist, or vehicle point as a Driveable Ground surface directly threatens path planning and collision avoidance modules by labeling occupied space as navigable road.

Under the COLA framework, intraclass confusions within a broad functional category (e.g., car versus truck under Vehicle) do not penalize the coarse metric ($mIoU_{\text{COLA}}$). Interclass confusions across primary categories (e.g., predicting a pedestrian point as road) remain strictly penalized.

The Ranking Inversion Paradox

Evaluating models on SemanticKITTI using both fine-grained ($mIoU_{\text{FG}}$) and coarse ($mIoU_{\text{COLA}}$) metrics reveals a clear discrepancy: the highest-ranking model on standard fine-grained metrics falls behind when evaluated on safety-critical coarse categories.

Architecture Category	Model	$mIoU_{\text{FG}}$ (%)	$mIoU_{\text{COLA}}$ (%)	Living Being Category Score (%)
Projection-based	FRNet	66.0	83.3	55.6
Sparse Conv-based	MinkowskiNet	64.3	87.3	83.5
Fusion-based	SPVCNN	65.3	86.8	83.2
Point-based	WaffleIron	65.8	87.4	83.2

Analytical Narrative

As reported in the paper’s empirical results, FRNet achieves the top fine-grained score on SemanticKITTI ($mIoU_{\text{FG}} = 66.0\%$). However, under the COLA taxonomy, its relative ranking drops ($mIoU_{\text{COLA}} = 83.3\%$), placing it substantially behind sparse convolution, point-based, and fusion-based methods.

This performance drop is driven by FRNet’s degraded performance on the Living Being category. FRNet achieves a score of 55.6% on Living Being, compared to 83.5% for MinkowskiNet, 83.2% for SPVCNN, and 83.2% for WaffleIron.

The authors attribute this vulnerability to the discretization mechanics of range-image projections. Projection-based architectures map 3D point clouds onto 2D spherical grids. Small, sparse target classes such as distant pedestrians or cyclists occupy very few point returns or grid pixels. When these sparse range images pass through standard convolutional downsampling layers, critical point features become diluted or merged with adjacent background elements (such as vegetation or static structures), causing interclass confusion between vulnerable road users and static surroundings. Consequently, high fine-grained benchmark accuracy can mask severe failures on safety-critical classes.

---

3. Pillar 2: Corruption Stress-Testing and Embedded Hardware Constraints

Corruption Taxonomy (Robo3D)

To analyze model degradation under adverse environmental and sensor conditions, the authors evaluated models on nuScenes-C and SemanticKITTI-C (derived from the Robo3D benchmark) across eight corruption types without fine-tuning:

• Atmospheric: Fog, Wet Ground, Snow (modeling environmental laser attenuation, backscatter, and road surface reflection).
• Geometric: Motion Blur, Beam Missing (modeling high-angular-velocity vehicle maneuvers and emitter hardware dropouts).
• Sensor: LiDAR Crosstalk, Incomplete Echo, Cross-Sensor (modeling multi-LiDAR pulse interference, low-reflectance target returns, and sensor hardware variations).

Model performance under corruption is quantified using the mean Corruption Error ($mCE$). Crucially, $mCE$ values are normalized against MinkowskiNet ($mCE = 100.0\%$). Scores below $100\%$ represent superior corruption resilience relative to the baseline, whereas scores above $100\%$ indicate greater vulnerability.

Hardware Realities and Latency Budgets

Evaluating corruption robustness in isolation is insufficient for physical deployment; robust perception models must also achieve hardware throughput that matches or exceeds the LiDAR sensor’s physical frame rate on embedded compute platforms.

To quantify this constraint, end-to-end runtime (data pre-processing plus network forward pass) was evaluated on an embedded NVIDIA Jetson AGX Orin system (2048-core Ampere GPU, 12-core Arm Cortex-A78AE CPU, 64 GB memory). Real-time operational thresholds were defined based on sensor acquisition frequencies:

• SemanticKITTI Threshold: $<100\text{ ms}$ per scan ($>10\text{ FPS}$ for the 10 Hz Velodyne HDL-64E).
• nuScenes Threshold: $<50\text{ ms}$ per scan ($>20\text{ FPS}$ for the 20 Hz Velodyne HDL-32E).

Model	Architecture Category	nuScenes-C Throughput (FPS)	SemanticKITTI-C Throughput (FPS)	Embedded Real-Time Pass/Fail
HARP-NeXt	Fusion-based	14.1	8.3	Near Real-Time (Fail 20 Hz / Pass 10 Hz Margin)
CENet	Projection-based	10.3	6.1	Fail
SalsaNext	Projection-based	—	9.2	Fail
MinkowskiNet	Sparse Conv-based	6.8	4.7	Fail
SPVCNN	Fusion-based	5.9	4.0	Fail
FRNet	Projection-based	2.6	2.5	Fail
WaffleIron	Point-based	1.9	0.5	Fail

Architectural Robustness Trade-offs

The paper’s cross-category evaluation demonstrates trade-offs across architectural families:

• Sparse Convolution (e.g., MinkowskiNet): Demonstrates strong resilience to geometric perturbations, securing the highest geometric corruption group average on nuScenes-C (70.9% mIoU). Spatial aggregation across non-empty 3D voxels limits structural sensitivity to motion blur and missing laser beams.
• Point-Based (e.g., KPConv, PTv3, WaffleIron): Continuous spatial aggregation yields high robustness to point-density loss. KPConv records the lowest mean Corruption Error on SemanticKITTI-C ($mCE = 99.5\%$) and the top atmospheric average (55.4%). However, point-based methods incur massive computational overhead on embedded platforms. For example, PTv3 achieves 4.1 FPS on a high-end desktop NVIDIA RTX 4090 GPU (and was omitted from Orin evaluation due to memory constraints), while WaffleIron drops from 9.0 FPS on an RTX 4090 down to 0.5 FPS on the Jetson AGX Orin (failing the 10 FPS threshold).
• Projection-Based (e.g., SalsaNext, CENet, FRNet): Offers higher inference speed on 2D range grids, but exhibits heterogeneous robustness. FRNet improves robustness over pure range-image baselines by retaining point features alongside projected representations ($mCE = 103.7\%$ on nuScenes-C; $99.5\%$ on SemanticKITTI-C), but its multi-representation overhead reduces execution speed to 2.5 FPS on the Jetson AGX Orin. Conversely, SalsaNext achieves 9.2 FPS on the Orin but suffers significant degradation under corruption ($mCE = 116.1\%$).
• Fusion-Based (e.g., HARP-NeXt): Delivers a balanced trade-off between corruption robustness and embedded throughput. HARP-NeXt achieves the highest atmospheric robustness group average on nuScenes-C (71.2% mIoU) and an overall $mCE$ of 105.7%, while maintaining an embedded execution throughput of 14.1 FPS on the Jetson AGX Orin.

---

4. Pillar 3: The Cross-Domain Generalization Wall

To evaluate zero-shot transferability, models trained on clean source datasets (nuScenes or SemanticKITTI) were evaluated directly on the unseen ParisLuco3D target dataset without domain adaptation or fine-tuning.

Shift Type 1: Appearance & Scene Shift (nuScenes 32-beam $\rightarrow$ ParisLuco3D 32-beam)

In this configuration, the source (nuScenes) and target (ParisLuco3D) datasets share identical sensor hardware (Velodyne HDL-32E), isolating environmental, geographic, and atmospheric variations without sensor-geometry confounding. ParisLuco3D introduces urban European layouts and wet-weather artifacts, including rain spray and standing water accumulation on road surfaces.

• Global Performance Degradation: Despite identical sensor hardware, top zero-shot performance across all evaluated models is capped at modest levels, led by MinkowskiNet (32.5% mIoU) and SPVCNN (32.1% mIoU). KPConv drops to 11.4% mIoU and Cylinder3D to 17.1% mIoU, reflecting sensitivity to wet-weather point cloud distortions.
• Catastrophic Rare-Class Failures: Concrete class-level evaluations from the paper’s data reveal near-total failure on rare or small-scale objects across almost all models:
  • Traffic Cone: Scores 0.0% IoU across every evaluated architecture.
  • Bicycle: Scores 4.7% IoU for CENet, 1.2% for FRNet, 1.1% for HARP-NeXt, 0.4% for MinkowskiNet, 0.1% for KPConv, and 0.0% for Cylinder3D.
  • Trailer: Scores 0.0% IoU across standard models, peaking at 1.7% for HARP-NeXt and KPConv.
  • Barrier: Scores remain low (MinkowskiNet at 1.8%, SPVCNN at 1.7%, CENet at 4.1%, KPConv at 4.2%).

Shift Type 2: Simultaneous Sensor & Scene Shift (SemanticKITTI 64-beam $\rightarrow$ ParisLuco3D 32-beam)

This setup evaluates transfer from SemanticKITTI (64-beam Velodyne HDL-64E) to ParisLuco3D (32-beam Velodyne HDL-32E). In addition to geographic shift, the vertical beam reduction alters scan density, point spacing, and angular resolution.

• Projection-Based Collapse: Range-image architectures relying on standard 2D ResNet backbones undergo near-total performance failure: CENet drops to 2.9% mIoU and FRNet drops to 3.4% mIoU. The authors explain that standard 2D ResNet backbones tightly couple spatial-channel feature learning directly to the fixed range-grid geometry and physical scan lines of the 64-beam training sensor. When evaluated on 32-beam inputs, this structural coupling causes complete spatial misalignment.
• Depth-Wise Separable Fusion Mitigation: In contrast to ResNet projection backbones, the fusion-based HARP-NeXt maintains 18.8% mIoU under the same shift. HARP-NeXt incorporates depth-wise separable convolutions within its projection processing path, which decouples spatial aggregation from channel-wise feature extraction, reducing spatial overfitting to specific beam layouts.
• Voxel and Continuous Geometry Transfer: Architectures built on 3D spatial representations demonstrate superior transferability under beam-count shifts: MinkowskiNet leads with 33.3% mIoU, followed by SPVCNN (31.5% mIoU), Cylinder3D (25.5% mIoU), and KPConv (22.1% mIoU). Continuous coordinate kernels (KPConv) and voxel encodings (MinkowskiNet, Cylinder3D) operate on local 3D geometric topology rather than fixed 2D range grids, making feature extraction more invariant to changing LiDAR beam densities.

---

5. Conclusion & Strategic Takeaways

The findings presented by Abou Haidar et al. establish that clean single-domain benchmark scores fail to predict real-world operational safety, corruption robustness, domain transferability, or embedded real-time viability. Models that dominate clean leaderboards can exhibit failures on safety-critical classes, collapse under minor beam-density shifts, or fail real-time embedded frame budgets.

Key Strategic Takeaways:

1. Clean Benchmark Rankings Mask Safety Vulnerabilities: Top fine-grained mIoU scores can mask severe performance drops in safety-critical categories like pedestrians and cyclists (Living Being), where projection-based downsampling leads to hazardous interclass confusion with driveable road surfaces.
2. Architectural Representations Dictate Robustness Bounds: Spatial voxel networks (e.g., MinkowskiNet) and continuous point kernels offer higher resilience to geometric corruptions and sensor beam shifts. Standard range-image projection backbones suffer catastrophic failure under domain shifts unless paired with channel-decoupled or multi-representation fusion mechanisms.
3. The Robustness-Efficiency Bottleneck Remains Unresolved: High-performing point-based architectures fail real-time operational latency budgets on embedded platforms like the NVIDIA Jetson AGX Orin (e.g., WaffleIron processing at 0.5 FPS versus the required 10 FPS threshold).
4. Zero-Shot Domain Generalization Is Unsolved: Direct zero-shot deployment to unseen environments results in severe performance loss, particularly for small or rare object classes. Developing deployment-ready perception stacks requires sensor-invariant architectural representations alongside corruption-aware training protocols.

Read the full paper on arXiv · PDF