ActiveSAM: Image-Conditional Class Pruning for Fast and Accurate Open-Vocabulary Segmentation

1. Introduction: The Bottleneck of Open-Vocabulary Segmentation

The Segment Anything Model 3 (SAM 3) has emerged as a formidable foundation for concept-prompted segmentation. However, a significant operational hurdle remains: applying these models to Open-Vocabulary Semantic Segmentation (OVSS) tasks is traditionally inefficient. Standard inference requires full-resolution decoding across an entire dataset vocabulary for every image—a massive waste of compute given that a typical scene contains only a fraction of potential classes.

ActiveSAM resolves this tension by serving as a training-free, zero-shot framework that converts SAM 3 into an “active-vocabulary” segmenter. Utilizing the Perception Encoder-Large+ (PE-L+) backbone, ActiveSAM achieves a superior speed-accuracy tradeoff without requiring weight updates or target-dataset training. By dynamically adapting its computational load to the specific semantic content of an image, it bridges the gap between foundation model power and real-world deployment needs.

2. The “Active Vocabulary” Breakthrough

The core innovation of ActiveSAM is Image-Conditional Class Pruning. Rather than processing an exhaustive list of categories, the model identifies an “active set” $A(I)$ of classes likely to be present before committing to high-resolution operations.

The inference process follows a sophisticated two-stage pipeline:

• Low-Resolution Presence Preview: The model executes a “cheap” preview at a resolution of $672 \times 672$. Critically, this stage skips the segmentation-head computation, utilizing only the frozen SAM 3 presence head to estimate class presence scores $q_c$. This gating mechanism allows the model to quickly filter the vocabulary with minimal latency.
• Full-Resolution Decoding: Only the retained classes in $A(I)$ are passed to the full-resolution ($1008 \times 1008$) grounding decoder. This ensures that the most expensive computational resources are reserved exclusively for relevant objects.

3. Four Pillars of Robust Perception

ActiveSAM’s performance is anchored by four technical modules that optimize both the lexical grounding and the spatial accuracy of the model.

3.1 Contextual Prompt Expansion (CPE)

Raw labels often lack the semantic density required for accurate alignment. CPE canonicalizes input strings through a lexical function $\eta$ (e.g., converting “windowpane” to “window pane”) and enriches them with WordNet hypernyms and vocabulary-level semantic neighbors. The final contextual prompt for a class $c$ is constructed as: $\bar{Z}_c = Z_c \parallel T^{nbr}_c \parallel T^{hyp}_c$ where $\parallel$ denotes concatenation along the token-sequence dimension. These expanded prompts are cached on the vocabulary-side, separate from the image-side evaluation, ensuring that semantic enrichment does not slow down per-frame inference.

3.2 Preview-driven Class Selection

To prevent performance degradation on simple scenes, ActiveSAM employs a gating threshold $V_{gate} = 40$. If the dataset vocabulary $V$ is larger than this threshold, the model generates an image-adaptive presence threshold $\tau(I)$ based on the presence scores $q_c$. This ensures the active set is sufficiently pruned for speed while remaining large enough to maintain high recall for the final grounding stage.

3.3 Bucketed Full-Resolution Decoding

ActiveSAM employs prompt multiplexing to accelerate the grounding decoder. Active classes are grouped into buckets of size $K=32$, allowing multiple prompts to share a single cross-attention pass. This significantly reduces the total number of decoder calls. Instance and semantic scores are fused via pixelwise maximization—$\max\{S^{inst}_c(p), S^{sem}_c(p)\}$—before the final labeling operation.

3.4 Margin-aware Background Calibration (MABC)

Standard background assignment usually relies on a simple threshold. MABC introduces a more principled approach by evaluating the margin between the top two class scores, $s_1(p)$ and $s_2(p)$. The calibrated confidence score is defined as: $c_{MABC}(p) = s_1(p) \sqrt{s_1(p) - s_2(p)}$ A pixel is assigned a semantic label only if it possesses a high score and a distinct margin from the runner-up; otherwise, it is calibrated as background, effectively suppressing false positives in complex scenes.

4. Performance Benchmarks: Speed and Accuracy

Benchmarks were conducted on a single NVIDIA RTX 5090, comparing ActiveSAM to the previous state-of-the-art (SegEarth-OV3) across eight standard OVSS benchmarks, including ADE20K and COCO-Stuff.

Metric	SegEarth-OV3 (Previous SOTA)	ActiveSAM (Ours)	Improvement
Mean mIoU (8 Benchmarks)	63.9	65.3	+1.4 mIoU
ADE20K mIoU	37.6	40.0	+2.4 mIoU
Inference Speed (Large Vocab)	1.0x	5.5x	5.5× Faster

On large-vocabulary datasets like COCO-Stuff (171 classes), ActiveSAM reduces the median number of active classes to approximately 29, yielding a massive reduction in latency without sacrificing segmentation fidelity.

5. Robustness: Failing Gracefully in the Real World

A “failure-first” analysis demonstrates that ActiveSAM is significantly more resilient to distribution shifts and image corruptions (ImageNet-C) than traditional CLIP-based baselines.

1. Noise Resilience: Under Gaussian noise and fog, ActiveSAM maintains a +1 to +2 mIoU lead over competitors.
2. Artifact Handling: The framework remains stable against motion blur and JPEG compression artifacts.
3. Architectural Stability: CLIP-based features often degrade rapidly under noise because they rely on high-frequency visual patch correlations. In contrast, SAM 3’s text-grounded decoder decouples recognition from localization. This allows the presence logic to remain stable even when the visual input is significantly degraded.

6. Deployment and Practical Implications for Embodied AI

For practitioners in robotics and autonomous driving, ActiveSAM provides a viable path for deploying foundation models on edge devices. By minimizing the prompt-decoding loop, it enables near-real-time performance in open-world environments.

Key Implementation Considerations:

• Recall Trade-offs: The reliance on a low-resolution ($672 \times 672$) preview for pruning introduces a safety trade-off. Exceptionally small or occluded objects may be missed during the preview stage, meaning they are never passed to the full-resolution decoder.
• Domain Limits: While CPE is highly effective for natural images via WordNet, specialized fields (e.g., medical or niche industrial robotics) may require custom lexical maps to maintain accuracy.

7. Conclusion: Key Takeaways

ActiveSAM effectively reconciles the computational intensity of foundation models with the agility required for real-world AI perception.

• Efficiency is Adaptive: By employing image-conditional decoding, the framework achieves up to 5.5× speedups in large-vocabulary environments.
• Training-Free is Viable: Superior OVSS performance is achievable using frozen foundation models (PE-L+), eliminating the need for costly weight updates.
• Context and Calibration Matter: Robust grounding requires both semantic enrichment (CPE) and principled background rejection (MABC) to handle the complexities of open-world scenes.

Read the full paper on arXiv · PDF