🚀 Accepted at ICSE 2026 (SEIP track), Rio de Janeiro, Brazil.
📄 Paper: link to paper

This repository contains the code, data, and experiments for our systematic study of how PDF parsing and text chunking choices affect end‑to‑end Retrieval‑Augmented Generation (RAG) performance on financial documents. It also hosts TableQuest, a new table‑focused QA benchmark built from real-world SEC filings and earnings reports.

Overview

This work presents the first systematic investigation of how key design choices in a Retrieval‑Augmented Generation (RAG) pipeline -PDF parsing and text chunking- impact end‑to‑end QA performance over financial documents, and provide actionable guidance for practitioners.

Key Contributions

• QA‑Centered Evaluation: We frame PDF understanding as a question‑answering task to mirror real analytical workflows.
• Benchmarks: We leverage two financial QA datasets, including TableQuest, our newly released table‑focused benchmark.
• Component Analysis: We compare multiple open‑source PDF parsers and six common chunking strategies (with varied overlap), and study their interactions.
• Practical Guidelines: Our results offer clear recommendations for building robust RAG systems on PDF corpora.

TableQuest targets QA over tables in financial documents (10‑K, 10‑Q, earnings reports). Each sample includes the PDF page(s) with tables plus QA pairs, organized by difficulty.

Source PDFs originate from the FinanceBench dataset.

The dataset is organized into three difficulty tiers based on the cognitive complexity required to answer the questions:

├── tablequest_dataset/            # TableQuest dataset
│   ├── annotation/                # Annotation and verification scripts for QA validation
│   │   ├── prompts/               # Prompt for automatic annotation tasks
│   │   ├── verification_openai/   # OpenAI-based verification
│   │   ├── analyze_vote_results.py
│   │   ├── openai_voter.py
│   │   └── README.md
│   ├── metadata/                  # Page and table metadata, sampling information
│   ├── prompts/                   # LLM prompt templates for different difficulty levels
│   ├── qa_pairs/                  # JSON files with question-answer pairs
│   ├── sampled_pages_pdf/         # Sampled pages organized by difficulty (PDF files)
│   ├── scripts/                   # Data processing and QA generation scripts
│   └── stats/                     # Statistics and analysis scripts/notebooks
├── src/                           # Evaluation framework
│   ├── chunkers/                  # Text chunking implementations
│   ├── evaluation/                # Evaluation metrics and classes (retrieval and answer correctness)
│   ├── generators/                # Answer generation modules
│   ├── parsers/                   # PDF parsing implementations
│   ├── preprocessing/             # Data preprocessing utilities
│   ├── retrievers/                # Information retrieval methods (BM25, dense, hybrid, ColBert)
│   ├── stats/                     # Stats utilities
│   └── test/                      # End-to-end testing and evaluation
├── requirements.txt               # Python dependencies
└── README.md                      # This file

# Clone the repository
git clone <repository-url>
cd finqa-tablequest-bench

# Create and activate Conda env
conda create -n finqa-tablequest python=3.11 -y
conda activate finqa-tablequest

# Install dependencies
pip install -r requirements.txt

Note: If you encounter issues with installation, you can comment out the 'Layout detection dependencies' section in requirements.txt, install the remaining packages, then uncomment and reinstall.

# Parse PDF documents into text
python src/parsers/text_parsers.py

# Chunk parsed text using different strategies
python src/chunkers/chunkers.py

# End-to-end indexing, retrieval & evaluation pipeline
python src/test/retrieval_evaluation.py

FinanceBench Dataset:

# Generate answers using Ollama models
python src/generators/ollama_answer_generation_financebench.py

# Generate answers using OpenAI models
python src/generators/openai_generation_financebench.py

TableQuest Dataset:

# Generate answers using Ollama models
python src/generators/ollama_answer_generation_tablequest.py

# Generate answers using OpenAI models
python src/generators/openai_generation_tablequest.py

# Answer correctness evaluation
python src/evaluation/llm_judge_evaluation_openai.py

• Parsers: PyPDF2, PyMuPDF, pdfplumber, pypdfium2, docling, unstructured
• Chunkers: token, sentence, semantic, recursive, SDPM, neural
• Retrievers: BM25, dense embeddings, ColBERT, SPLADE