DR$^{3}$-Eval: Towards Realistic and Reproducible Deep Research Evaluation

1. Introduction: The Dawn of the Autonomous Researcher

We are witnessing a fundamental phase shift in Artificial Intelligence. The era of the “helpful chatbot” is giving way to the age of the Deep Research Agent (DRA). Unlike their predecessors, these agents don’t just answer questions; they inhabit the role of a digital scholar—independently planning complex workflows, navigating vast information ecosystems, and synthesizing high-fidelity, citation-grounded reports.

However, this newfound autonomy has triggered a profound “evaluation crisis.” Researchers are currently forced to choose between two flawed testing grounds: the live web, which is plagued by temporal volatility (where search results fluctuate by the hour), and “clean” local datasets that lack the messy, contradictory nature of real-world research. DR³-Eval enters the fray as the first benchmark to reconcile these extremes, providing a “Truth Sandbox” that is as realistic as the open web but as controllable as a laboratory.

2. The Reproducibility Trap: Why Live-Web Testing Fails

To build better agents, we must first be able to measure them reliably. Current benchmarks like Deep Research Bench offer ecological validity through live web access, but they fail the reproducibility test; a model’s success might depend on a search engine’s algorithm on a specific Tuesday rather than its inherent reasoning. Conversely, frameworks like DRBench lack the “noise” of the real world, while even advanced sandbox systems like DeepResearchGym often simplify the environment to text-only queries.

DR³-Eval fills these critical gaps by addressing the three pillars of modern research:

• Multimodal Grounding: Agents must interpret everything from CSVs and PDFs to audio and video.
• Noise-Intensive Environments: Simulating the distractions, biases, and outdated information of the open web.
• Verifiable Solution Paths: Ensuring that every complex task has a singular, reachable “ground truth.”

3. Inside the Sandbox: Simulating the Chaos of the Open Web

The cornerstone of DR³-Eval is the Static Sandbox Corpus. Rather than letting an agent roam the unpredictable live web, it is placed in a bespoke, verifiable environment containing an average of 465.5 web pages per task. To build this “digital haystack,” the researchers utilized a “Divergent-Convergent” methodology:

• The Divergent Stage: Like a detective casting a wide net, Gemini-2.5-Pro brainstorms a diverse array of keywords to simulate a broad brainstorming session.
• The Convergent Stage: The model then zooms in on the “smoking gun,” categorizing keywords into “Signal” (evidence-rich paths) and “Noise” (distractions).

This process ensures that agents must exercise critical judgment to find the truth. The rigor of this environment is underscored by a 35.7% task pass rate—only the most unambiguous, high-purity tasks survive the quality control funnel.

The Anatomy of a Sandbox

Document Category	Description
Supportive Web Pages	Manually verified results providing necessary and sufficient evidence to solve the query.
Distractor Web Pages	Seemingly relevant but confirmed to be outdated, one-sided, or inaccurate content.
Noise Web Pages	Thematically related results designed to test the agent’s ability to ignore irrelevant info.

4. The Multimodal Challenge: Beyond Just Text

Real-world research is a multimodal marathon. DR³-Eval reflects this by incorporating a 50/50 split of English and Chinese tasks across 13 domains, from Finance to Healthcare. The benchmark’s data distribution is a testament to its complexity:

• 45.98% Documents (PDFs, Word, PPT)
• 27.68% Images (PNG, JPEG, WebP)
• 13.84% Videos (MP4)
• Remainder: Audio (MP3), CSV/Excel, and HTML files.

A critical innovation here is Reverse Construction. Rather than asking open-ended questions, researchers start with verified evidence and work backward to create a query. This eliminates “shortcut solutions”—tasks that could be solved by a simple Google search—and forces the agent to navigate the “Long-tail effect” of information quality by combining user files with sandbox data. A “leave-one-out” verification process ensures the task is strictly impossible to solve without specific evidence from the sandbox.

5. Measuring Brilliance: The 5-Dimensional Report Card

DR³-Eval utilizes a multi-dimensional framework to judge agents, with GPT-5.1 serving as the lead judge for text and Gemini-2.5-Pro assisting with the verification of claims grounded in video and audio content.

• Information Recall (IR): The percentage of specific insights captured from both user files and the sandbox.
• Factual Accuracy (FA): A rigorous verification of claim-source pairs to identify hallucinations.
• Citation Coverage (CC): Described as an “irreplaceable literature metric,” this measures the agent’s ability to identify the documents strictly necessary for the query.
• Instruction Following (IF): Adherence to a custom-generated checklist of content and formatting requirements.
• Depth Quality (DQ): An expert-level assessment of analytical substance and logical rigor.

6. The Reality Check: Key Takeaways from the Benchmarking Trials

To test the benchmark, researchers developed the DR³-Agent, a system built on the MiroFlow framework. Unlike traditional RAG systems that use a simple “Top-K” lookup, the DR³-Agent employs a ReAct-based Agentic RAG paradigm, performing autonomous, multi-step retrieval and iterative query refinement.

Testing state-of-the-art models like Claude Sonnet 4, GPT-4.1, and Gemini-2.5-Pro revealed three jarring truths:

1. The “Noise Floor” Effect: As the sandbox context length grows from 32k to 512k tokens, performance craters. Even the best models struggle to distinguish signals once the “haystack” reaches a certain volume.
2. The Hallucination Paradox: Models like Qwen3-235B-A22B and GPT-4.1 often score high on “Instruction Following” while suffering from abysmal “Factual Accuracy.” They produce reports that look authoritative and structured but are built on a foundation of false information.
3. Scaling Law Limits: While larger models generally lead, the benchmark remains “highly challenging” for all, proving that raw parameters cannot replace sophisticated reasoning in noisy environments.

In these trials, Claude Sonnet 4 emerged as the most resilient researcher, though still far from a perfect score.

7. Conclusion: Toward More Trustworthy AI Researchers

DR³-Eval is a vital contribution to AI safety and red-teaming. By stripping away the shield of live-web volatility, it exposes “systematic and covert” failure modes—retrieval robustness issues and “confident” hallucinations—that traditional evaluations miss. As we delegate more high-stakes analytical work to autonomous agents, we must ensure they possess the critical judgment to navigate a world where information is plentiful, but truth is often buried.

Final Takeaway: The next frontier of AI is not the ability to retrieve information, but the ability to exercise critical judgment in an environment saturated with noise and misinformation.

Read the full paper on arXiv · PDF