MLLM-Routed Heterogeneous Ensembles for Robust Cross-Dataset Image Classification

1. Introduction: The Cross-Domain Generalization Wall

Modern image classification models achieve exceptional performance when trained and evaluated on isolated, task-specific datasets. However, real-world vision systems—such as those embedded in autonomous robotics, medical diagnostics, and mobile AI assistants—must operate across diverse visual environments. When exposed to broad cross-dataset distribution shifts (e.g., transitioning between natural photographs, diagnostic medical scans, subtle human facial expressions, and high-altitude satellite imagery), single-architecture classifiers frequently suffer severe performance degradation.

In benchmark evaluations conducted by Perkins et al., this fragility highlights fundamental trade-offs across conventional deep learning paradigms:

• Convolutional Neural Networks (CNNs): Exemplified by architectures like ResNet-50, CNNs excel at capturing localized textures, fine edges, and spatial hierarchies. However, their fixed inductive biases make them sensitive to distribution shifts, frequently resulting in cross-domain overfitting when forced to evaluate out-of-distribution visual inputs.
• Self-Supervised Learning (SSL): Attention-based representation learners, such as DINOv2 and DINOv3, extract structural visual features without relying on dense human annotations. While SSL backbones demonstrate broad transferability, fine-tuning a single backbone on a specialized downstream task often restricts the model’s feature space, degrading performance across disjoint visual domains.
• Vision-Language Models (VLMs): Contrastive architectures like CLIP map images and natural language into a shared embedding space, enabling broad zero-shot alignment and scene-level semantic understanding. However, VLMs often exhibit reduced accuracy when evaluating fine-grained, highly specialized anatomical or domain-specific textures.

To mitigate single-backbone vulnerabilities, machine learning engineers traditionally deploy ensemble architectures. Dense fusion ensembles concatenate representations or average output probabilities across multiple backbones, but they incur high computational overhead by executing every constituent model for every input frame. Learned dynamic routing frameworks—such as Expert Gate or Neural Logit Controllers (NLC)—route inputs to specific models to reduce inference latency. However, traditional routers suffer from high structural rigidity: introducing a new task or domain requires re-architecting and retraining the entire meta-classifier. Furthermore, learned neural routers operate as opaque black boxes, offering safety researchers and auditors no semantic visibility into why a specific backbone was selected for a given input.

To address these operational and safety limitations, Perkins et al. introduced ARMDIL (Adaptive Router for Multi-Domain Image Classification with LLMs). ARMDIL replaces rigid, black-box meta-classifiers with a zero-shot, prompt-driven Multimodal Large Language Model (MLLM) agent. By combining high-level visual reasoning with dynamic expert selection, ARMDIL provides an interpretable and adaptable alternative to traditional trained neural ensembles.

---

2. Deconstructing the ARMDIL Architecture

The ARMDIL framework routes incoming images to heterogeneous domain experts through a multi-stage operational pipeline orchestrated by an MLLM agent.

                  [Input Image]
                        │
        ┌───────────────┴───────────────┐
        ▼                               ▼
 [Image Preprocessing]     [Auxiliary Quality Metrics]
 (Resize & Normalize)      • Perceptual Blur Score
                           • Mean Brightness
                           • Grayscale Contrast
                           • Wavelet Noise MAD
        │                               │
        └───────────────┬───────────────┘
                        ▼
         [Gemma-4-12B MLLM Router Agent]
       (Visual Semantics + CoT Reasoning)
                        │
  ┌─────────────┬───────┴───────┬─────────────┬─────────────┐
  ▼             ▼               ▼             ▼             ▼
`GENERAL`    `FACIAL`      `GEOGRAPHIC`   `MEDICAL`     `UNSURE`
  │             │               │             │             │
(DINOv2)     (DINOv3)        (CLIP)      (ResNet-50) (Balanced DINOv3)
  │             │               │             │             │
  └─────────────┴───────┬───────┴─────────────┴─────────────┘
                        ▼
          [Unified N=38 Head Output]

Operational Pipeline

1. Preprocessing & Quality Signal Generation: The input image is resized and normalized. Simultaneously, an automated image-quality assessment pipeline calculates four explicit heuristic metrics derived from established signal-processing algorithms:
   • Blur score: Evaluated using the no-reference perceptual blur metric introduced by Crété-Roffet et al. (2007), rating perceptual blur on a continuous scale from 0.0 (sharp) to 1.0 (blurry).
   • Mean brightness: Calculated as the mean intensity of image pixels following grayscale conversion (0.0 = pure black, 1.0 = pure white).
   • Contrast: Derived from the standard deviation of grayscale pixel intensities (0.0 = extremely low contrast, 0.25+ = very high contrast).
   • Noise estimate: Computed using Donoho & Johnstone’s (1994) median absolute deviation (MAD) of wavelet detail coefficients (0.0 = noise-free, 0.10+ = highly noisy).
2. Unified Label Space Construction: All constituent expert models are fine-tuned with a shared, unified $N=38$ classification head that aggregates class labels across four diverse public datasets: $\text{CIFAR10 } (10 \text{ classes}) + \text{FER2013 } (7 \text{ classes}) + \text{EuroSAT } (10 \text{ classes}) + \text{OrganAMNIST } (11 \text{ classes}) = 38 \text{ total logits}$ Structuring all linear heads over this joint, disjoint label space ensures that even if an image is routed to a sub-optimal expert, the selected model outputs a valid, mathematically well-formed logit distribution across the entire benchmark label space.
3. Agent Decision Engine Prompting: The raw visual input and its auxiliary quality statistics are provided to a quantized local MLLM router (Gemma-4-12B UD-Q5). Using Chain-of-Thought (CoT) reasoning, the MLLM evaluates visual concepts alongside heuristic statistics to categorize the image into one of five domain aliases: GENERAL, FACIAL, GEOGRAPHIC, MEDICAL, or UNSURE.
4. Dynamic Expert Execution & Scapegoat Routing: The MLLM delegates classification to the corresponding domain expert backbone. If the MLLM router exhibits ambiguity, it outputs the UNSURE alias, which redirects the sample to the generalist baseline model possessing the highest overall validation accuracy: Balanced DINOv3 (ViT-L/16).

Multi-Domain Dataset Mapping

Perkins et al. structured their multi-domain benchmark using four public image datasets, mapping each dataset to an abstract domain alias within the MLLM system prompt:

Benchmark Dataset & Assigned Alias	Visual Description & Target Scope
CIFAR10 GENERAL	Natural, object-centric images encompassing animals, vehicles, and everyday items across 10 balanced categories.
FER2013 FACIAL	Close-up human faces, portraits, and expression crops categorized across seven distinct emotional states.
EuroSAT GEOGRAPHIC	Multi-spectral high-altitude satellite imagery, land-cover perspectives, and aerial terrain environments across 10 classes.
OrganAMNIST (O-MNIST) MEDICAL	Diagnostic 2D medical scans, pathology cross-sections, and anatomical CT slices covering 11 human organ categories.

---

3. Architecture Vulnerabilities & Expert Specialization

To select domain experts for ARMDIL, Perkins et al. evaluated candidate backbones across three architectural families trained under domain-skewed (70% target domain bias) and balanced distributions. Their empirical findings highlight distinct capability boundaries and failure modes for each backbone type.

       [Architectural Capability & Specialization Profile]

       ResNet-50 (CNN) ──────────► Fine Texture & Local Edges
                                   (Peak: Medical Scans / OrganAMNIST)

       DINOv2/v3 (SSL) ──────────► Structural & Feature Stability
                                   (Peak: Facial Expressions & Objects)

       CLIP ViT-L/14* (VLM) ─────► Scene-Level Semantic Alignment
                                   (Peak: Satellite Terrain / EuroSAT)

       *Note: Text describes ViT-L/14 while paper tables list ViT-L/16.

• ResNet-50 (Convolutional Neural Networks): In the benchmark evaluations conducted by Perkins et al., ResNet-50 demonstrated high parameter efficiency and strong performance when extracting localized features, fine anatomical edges, and high-frequency visual textures. As a result, ResNet-50 achieved the top classification performance on diagnostic medical scans (97.20% top-1 accuracy on OrganAMNIST). However, ResNet-50 exhibited marked sensitivity to distribution shifts and a tendency toward cross-domain overfitting when trained on multi-domain data.
• DINOv2 & DINOv3 (Self-Supervised Learning): Attention-based representation learners displayed superior structural stability and broad representation capacity. DINOv3 (ViT-L/16) achieved state-of-the-art results on nuanced facial emotion recognition (70.49% top-1 accuracy on FER2013), outperforming all competing backbones. DINOv2 (ViT-L/14) achieved top performance on natural object classification (99.24% top-1 accuracy on CIFAR10).
• CLIP ViT-L/14 + LoRA (Vision-Language Models): Parameter-efficient fine-tuning via LoRA adapters inserted into OpenCLIP ViT-L/14 enabled strong scene-level semantic alignment. CLIP outperformed all candidate architectures on high-altitude aerial terrain (98.56% top-1 accuracy on EuroSAT), benefiting from broad conceptual representations rather than localized texture extraction. (Note: Perkins et al. report using OpenCLIP ViT-L/14 in Section 3.2.3 of their paper, while labeling the model as CLIP ViT-L/16 in Table 2; both designations refer to the same VLM expert within their experimental setup).

Backbone Capability & Vulnerability Comparison

The empirical findings reported in Tables 2 and 3 of the paper are summarized below:

Architectural Family	Core Strengths	Primary Vulnerabilities	Peak Target Domain Performance
ResNet-50 (CNN)	High parameter efficiency; precise local edge and high-frequency texture extraction.	Sensitive to distribution shift; prone to cross-domain feature overfitting.	MEDICAL (OrganAMNIST: 97.20% Acc / 96.97% F1)
DINOv2 / DINOv3 (SSL)	Attention-driven structural representations; high resilience under shift.	Greater memory foot-print; higher computational cost during downstream adaptation.	FACIAL (FER2013: 70.49% Acc / 67.75% F1) & GENERAL (CIFAR10: 99.24% Acc / 99.28% F1)
CLIP ViT-L/14 (VLM)*	Global scene-level semantic alignment; strong domain resilience.	Lower accuracy on fine-grained, localized anatomical textures.	GEOGRAPHIC (EuroSAT: 98.56% Acc / 98.51% F1)

---

4. Empirical Evaluation: ARMDIL vs. Traditional Routers & Baselines

Perkins et al. benchmarked ARMDIL against individual backbones, an unweighted Majority Vote (MV) ensemble, a fully trained ResNet-18 Neural Network Router (NN-Router), and an ideal theoretical “Oracle” router (which assumes 100% domain routing accuracy).

Benchmark Performance Comparison

Performance metrics across individual datasets and the global unified dataset are detailed below:

Method / Router Paradigm	CIFAR10 Acc / F1 (%)	EuroSAT Acc / F1 (%)	FER2013 Acc / F1 (%)	OrganAMNIST Acc / F1 (%)	Overall Acc / Macro F1 (%)
DINOv3 (Balanced Backbone)	99.05 / 99.06	97.24 / 97.19	69.82 / 66.63	92.52 / 92.22	89.61 / 89.52
Majority Vote Ensemble (MV)	98.87 / 98.87	98.42 / 98.38	68.35 / 64.64	96.25 / 95.84	90.47 / 89.43
NN-Router (ResNet-18 Meta-Classifier)	99.13 / 99.18	98.56 / 98.52	69.81 / 66.98	97.22 / 96.95	91.18 / 90.41
ARMDIL (Gemma-4-12B MLLM Router)	99.02 / 99.05	97.91 / 97.88	70.49 / 67.73	96.69 / 96.38	90.78 / 90.71
Theoretical Oracle Router	99.24 / 99.28	98.56 / 98.51	70.49 / 67.75	97.20 / 96.97	91.04 / 90.99

Key Analytical Insights

• Overall Accuracy & F1 Performance: ARMDIL achieved 90.78% top-1 overall accuracy and a 90.71% global macro F1 score across the unified test dataset. This performance surpasses the best individual non-routed backbone (balanced DINOv3 at 89.61%) by 1.17% in accuracy and outperforms the Majority Vote baseline by 0.31% in accuracy and 1.28% in macro F1.
• Analytical Nuance in Macro F1 vs. Top-1 Accuracy: While the trained 4-class ResNet-18 NN-Router achieved slightly higher overall top-1 accuracy (91.18% vs. 90.78%), ARMDIL outperformed the NN-Router on overall macro F1 score (90.71% vs. 90.41%). This occurs because macro F1 treats all four evaluation domains equally. ARMDIL achieved near-perfect routing accuracy (99.82% true positive rate) on FER2013—the most difficult and class-skewed domain—where it yielded a dataset F1 score of 67.73% (compared to 66.98% for the NN-Router). Because global macro F1 averages per-dataset F1 scores equally across all four domains, ARMDIL’s superior accuracy on the hardest dataset elevated its overall macro F1 score above the NN-Router, despite minor routing hesitations on satellite imagery.
• Zero-Shot Routing vs. Dedicated Meta-Classifiers: As reported in Appendix A.2.2 of the paper, the ResNet-18 NN-Router achieved a 99.5% domain classification accuracy across the four domains after dedicated supervised training. In contrast, Gemma-4-12B achieved a zero-shot average domain routing accuracy of 92.76% (Table 6). Despite receiving zero domain-routing training, ARMDIL maintained performance parity with the supervised NN-Router, staying within 0.40% of its top-1 classification accuracy.
• Routing Confusion Matrix Analysis: The Gemma-4-12B routing matrix (Table 4 in the paper) demonstrated high true-positive routing accuracy for facial portraits (99.82%), natural objects (97.80%), and medical scans (95.21%). In contrast, EuroSAT satellite imagery exhibited lower routing confidence (78.20% true positive rate, with 12.72% categorized as UNSURE). Perkins et al. observed that texture-uniform aerial imagery often lacks prominent object boundaries, causing MLLM routing hesitation.
• Ablation Findings (Self-Consistency & Quality Metrics): Ablation experiments (Tables 6 and 7) evaluated multi-prompt self-consistency (SC) without Chain-of-Thought reasoning. Self-consistency improved EuroSAT routing accuracy (increasing domain accuracy to 82.17%, and 83.87% when quality metrics were omitted) by filtering out sporadic UNSURE classifications. Furthermore, omitting image quality metrics proved beneficial for satellite imagery because uniform spatial textures were frequently mischaracterized by heuristic algorithms as lens blur.

---

5. Interpretability & Adaptability: The Safety Dividend

Beyond competitive classification performance, MLLM-driven dynamic routing provides distinct advantages for AI safety, model auditing, and system maintainability.

Natural Language Audit Traces

Traditional neural routers and dynamic logit controllers operate as opaque numerical vectors, concealing the internal logic behind expert selection. ARMDIL addresses this opacity by generating human-readable Chain-of-Thought (CoT) reasoning traces for every routing decision. These traces allow safety researchers and red-teamers to inspect visual feature extractions, evaluate heuristic contributions, and audit failure boundaries.

Sample MLLM Reasoning Trace (OrganAMNIST Left Kidney Scan):

The user wants me to classify an image into one of five categories: GENERAL, MEDICAL, GEOGRAPHIC, FACIAL, or UNSURE.

1. Analyze the image: The image shows a grayscale, cross-sectional view of what appears to be an internal body part. It looks like a CT scan or MRI of a lung or chest area. There are distinct anatomical structures, including what looks like lung tissue (darker areas) and soft tissue/vessels (lighter areas). This fits the definition of MEDICAL imagery (scans, pathology, etc.).
2. Analyze the statistics: blur score: 0.53 (somewhat blurry, common in medical scans); mean brightness: 0.38 (mid-dark); contrast: 0.20 (high contrast, typical for medical imaging to distinguish tissue); noise estimate: 0.0011 (low noise).
3. Compare with definitions: GENERAL: Not everyday objects. MEDICAL: Matches (CT/MRI scan). GEOGRAPHIC: Not satellite/aerial. FACIAL: Not a human face. UNSURE: The image is clearly a medical scan.
4. Final Decision: MEDICAL.

Safety Takeaway: Although the MLLM misidentified the specific organ (mistaking a kidney for a lung), its high-level domain reasoning remained robust enough to select the MEDICAL domain expert, which subsequently produced the correct downstream organ classification.

Prompt-Driven Adaptability

In conventional ensemble architectures, integrating a new image domain or introducing an updated backbone requires retraining meta-classifiers or re-estimating temperature fusion weights across the entire dataset.

ARMDIL eliminates this requirement. To integrate a new task—such as adding a VEHICLES category for autonomous driving applications—an operator appends a semantic domain definition to the system prompt and registers the corresponding expert model. This prompt-driven setup enables system expansion without fine-tuning or retraining routing parameters.

---

6. Conclusion & Practical Takeaways

Perkins et al. demonstrate that multimodal LLM agents can orchestrate heterogeneous vision backbones across diverse visual domains. By replacing black-box meta-classifiers with agentic routing, ARMDIL achieves dynamic model selection while providing transparent, human-readable reasoning traces.

Actionable Takeaways for AI Safety & Vision System Architects

• Zero-Shot Routing Performance: MLLM agentic routing achieves performance parity with specialized, fully trained neural meta-classifiers (achieving a 90.71% global macro F1 vs. 90.41% for an NN-Router) without requiring router training or dataset-specific optimization.
• Auditable Reasoning Traces: Chain-of-Thought output traces provide built-in decision transparency, allowing safety practitioners to audit failure boundaries and inspect routing logic.
• Zero-Retraining System Expansion: New domain experts and target task categories can be integrated into production pipelines via system prompt updates, bypassing the costs of meta-classifier retraining.
• Awareness of Heuristic Edge Cases: Low-level perceptual quality metrics (e.g., blur and contrast scores) can confuse MLLM routers when evaluating domain-uniform textures such as aerial satellite imagery. Prompt configurations should account for these domain-specific characteristics.
• Scalability: Future directions identified by the authors include deploying larger commercial MLLMs to enhance reasoning accuracy, refining system prompts, and expanding evaluations to broader multi-domain benchmarks.

Read the full paper on arXiv · PDF