Abstract
Multimodal large language models (MLLMs) achieve strong performance on benchmarks that evaluate text, image, or video understanding separately. However, these settings do not assess a critical real-world requirement, which involves retrieving relevant evidence from large, heterogeneous multimodal corpora prior to reasoning. Most existing benchmarks restrict retrieval to small, single-modality candidate sets, substantially simplifying the search space and overstating end-to-end reliability. To address this gap, we introduce MultiHaystack, the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions. MultiHaystack comprises over 46,000 multimodal retrieval candidates across documents, images, and videos, along with 747 open yet verifiable questions. Each question is grounded in a unique validated evidence item within the retrieval pool, requiring evidence localization across modalities and fine-grained reasoning. In our study, we find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus. Additionally, even the strongest retriever, E5-V, achieves only 40.8% Recall@1, while state-of-the-art MLLMs such as GPT-5 experience a significant drop in reasoning accuracy from 80.86% when provided with the corresponding evidence to 51.4% under top-5 retrieval. These results indicate that multimodal retrieval over heterogeneous pools remains a primary bottleneck for MLLMs, positioning MultiHaystack as a valuable testbed that highlights underlying limitations obscured by small-scale evaluations and promotes retrieval-centric advances in multimodal systems.
Examples of six tasks in MultiHaystack: Visual Parsing & Positioning (spatial layouts), Contextual Understanding (embedded text), Video Temporal Reasoning (motion/order), Statistical Reasoning (charts/tables), Metadata Identification (affiliations/timestamps), and Factual Knowledge Retrieval (corpus-grounded facts).
Benchmark construction pipeline. MultiHaystack is built in four stages: collecting diverse multimodal sources, generating specific QA pairs, filtering for unique and grounded answers, and enriching with data. This design ensures coverage across six tasks (Fig. 3) and overcomes the unimodal, small-scale, or ambiguous limitations of prior benchmarks.
Error cases illustrating retrieval errors, including modality bias (retrieving images instead of video evidence) and semantic drift (violating temporal constraints), and reasoning errors, including visual numeracy (misreading numbers in charts) and layout-aware multi-step reasoning (failing to integrate structured cues across layouts).
Performance on MultiHaystack. "Gold in Top-1/5" directly provides answer-containing files; "Single-Modality" and "Cross-Modality" require retrieval within one or across multiple modalities.
Top-k ablation analysis for MLLMs integrated with E5-V.
Comparison in three distinct modalities. (a) represents the video modality, (b) represents the image modality, and (c) represents the document modality.
Pool-size controlled comparison. Recall of MM-Embed under single-modality retrieval and mixed-modality retrieval with an identical total pool size. Performance remains substantially lower in the mixed-modality condition, indicating that the cross-modal gap arises from modality heterogeneity rather than pool size.
Effect of data enrichment under varying candidate pool sizes, showing that recall consistently drops as the pool expands.
BibTeX
@article{YourPaperKey2024,
title={Your Paper Title Here},
author={First Author and Second Author and Third Author},
journal={Conference/Journal Name},
year={2024},
url={https://your-domain.com/your-project-page}
}