This benchmark evaluates how effectively different PDF parsing solutions extract mathematical formulas and tables from documents. We generate synthetic PDFs with diverse formatting scenarios, parse them with different parsers, and score the extracted content using LLM-as-a-Judge. This semantic evaluation approach substantially outperforms traditional metrics in agreement with human judgment.

Results are based on two separate benchmark datasets, each containing 100 synthetic PDFs:

• 2026-q1-tables-only — PDFs with tables of varying complexity (simple, moderate, complex)
• 2026-q1-formulas-only — PDFs with inline and display-mode mathematical formulas

Legend:

• Tables: Average LLM-as-a-Judge score (0-10) across 451 tables from 2026-q1-tables-only
• Formulas: Average LLM-as-a-Judge score (0-10) across 1413 inline + 657 display formulas from 2026-q1-formulas-only
• Cost/Time: API cost (USD) for 100 pages, or GPU wall-clock time on a single NVIDIA RTX 4090
• Inference: Deployment options (CPU, GPU, API)

Rule-based metrics correlate poorly with human judgment for formula and table extraction. We validated this in two human annotation studies (Pearson r = correlation with human scores):

• formula-metric-study — 750 human ratings: text metrics reach r = 0.01, CDM r = 0.31, LLM judges r = 0.74–0.82
• table-metric-study — 1,500+ human ratings: rule-based metrics (TEDS, GriTS) top out at r = 0.70, LLM judges reach r = 0.94

PDFs are generated synthetically using LaTeX with randomized parameters (document class, fonts, margins, column layout, line spacing) to test parser robustness across different formatting scenarios. Since PDFs are generated from LaTeX source, we automatically obtain exact ground truth as a byproduct of the generation process.

• Formula Dataset: Each PDF contains randomly selected formulas embedded in text passages, displayed as inline or display-mode equations. Formulas are sampled from our dataset of 319,000 formulas extracted from Wikipedia, ensuring diversity in complexity and real-world relevance. Dataset: piushorn/wikipedia-latex-formulas-319k

• Table Dataset: Each PDF contains tables of varying complexity (simple, moderate, complex) with diverse content types, column layouts, and formatting. Dataset coming soon on Hugging Face.

Parser outputs are assessed using a two-step pipeline:

Given a parser's output (the extracted text from a PDF), an LLM establishes initial correspondences between extracted elements and ground truth (formulas or tables), then fuzzy search reliably extracts exact strings from the parsed output. This achieves robust alignment even when parser outputs differ significantly from ground truth.

The primary metric is the LLM-as-a-Judge score (0-10 scale, default: Gemini 3 Flash via OpenRouter). For formulas, the judge evaluates correctness, completeness, and semantic equivalence. For tables, the judge evaluates content accuracy and structure preservation. Scores are computed separately for inline/display formulas and by table complexity (simple/moderate/complex). See Why LLM-as-a-Judge? for our validation studies.

Benchmark Datasets: New benchmark datasets are released quarterly, each containing 100 PDFs. The current datasets are 2026-q1-tables-only (tables) and 2026-q1-formulas-only (formulas).

There are two ways to use this benchmark, depending on your needs:

Use this if: You quickly want to evaluate your PDF Parsing tool against the benchmark.

Advantage: Simple pip install, no need to integrate with the repository structure.

pip install pdf-parse-bench

Note: Set OPENROUTER_API_KEY environment variable for evaluation (used for LLM-as-a-Judge scoring via OpenRouter).

Get the benchmark PDFs and parse them with your parser:

from pdf_parse_bench import get_benchmark_pdfs_dir
from pathlib import Path

# Get benchmark PDFs (included in the package)
pdfs_dir = get_benchmark_pdfs_dir()

# Parse each PDF with your parser
output_dir = Path("results/my_parser")
for pdf_path in pdfs_dir.glob("*.pdf"):
    # Parse PDF with your parser
    parsed_text = your_parser.parse(pdf_path)

    # Save to expected format: {output_dir}/{pdf_name}/parsed.md
    (output_dir / pdf_path.stem / "parsed.md").parent.mkdir(parents=True, exist_ok=True)
    (output_dir / pdf_path.stem / "parsed.md").write_text(parsed_text)

Required output structure:

results/my_parser/
├── 000/
│   └── parsed.md
├── 001/
│   └── parsed.md
├── 002/
│   └── parsed.md
...

Run the benchmark evaluation on your parsed results:

from pathlib import Path
from pdf_parse_bench import Benchmark, PDFParser, get_benchmark_ground_truth_dir

class MyParser(PDFParser):
    @classmethod
    def display_name(cls) -> str:
        return "My Parser"

    def parse(self, pdf_path: Path, output_path: Path) -> str:
        raise NotImplementedError  # Already parsed in Step 1

bench = Benchmark(
    parser_output_dir=Path("results/my_parser"),
    ground_truth_dir=get_benchmark_ground_truth_dir(),
    llm_judge_models=["google/gemini-3-flash-preview"],
    parser=MyParser(),
)
bench.extract()
bench.evaluate()
bench.save_benchmark_summary()

Use this if: You want to contribute your parser to the benchmark, reproduce published results, or ensure full reproducibility of your evaluation setup.

Advantage: Full automation with CLI, parser configuration is versioned and reproducible, easy to share exact setup with others.

git clone https://github.com/phorn1/pdf-parse-bench.git
cd pdf-parse-bench

# Install with uv
uv sync

# Configure environment (copy and edit .env.example)
cp .env.example .env

Create a new parser module in the parsers/ directory:

# parsers/my_parser/__main__.py
from pathlib import Path
from pdf_parse_bench.utilities import PDFParser
from pdf_parse_bench.pipeline import run_cli

class MyParser(PDFParser):
    @classmethod
    def display_name(cls) -> str:
        return "My Parser"

    def parse(self, pdf_path: Path, output_path: Path) -> str:
        # Your parsing logic here
        markdown = "# Parsed content"
        self._write_output(markdown, output_path)
        return markdown

if __name__ == "__main__":
    run_cli(MyParser())

uv run -m parsers.my_parser -i data/<dataset>

The benchmark infrastructure handles everything automatically:

• Loading test PDFs from data/<dataset>/pdfs/
• Parsing each PDF with your parser
• Extracting formulas from parsed output
• Running evaluation against ground truth
• Saving results in standardized format

The benchmark CLI provides several options to customize execution:

# Run only specific steps
uv run -m parsers.my_parser -i data/<dataset> --step parse
uv run -m parsers.my_parser -i data/<dataset> --step extract --step evaluate

# Reprocess existing results
uv run -m parsers.my_parser -i data/<dataset> --reprocess all
uv run -m parsers.my_parser -i data/<dataset> --reprocess parse --reprocess extract

# Use a different LLM judge for evaluation (OpenRouter model format)
uv run -m parsers.my_parser -i data/<dataset> --llm-judge-model openai/gpt-5-mini

pdf-parse-bench/
├── src/pdf_parse_bench/       # Core benchmark infrastructure
│   ├── pipeline/              # Benchmark execution pipeline
│   ├── eval/                  # Evaluation metrics and judges
│   ├── extraction/            # Formula extraction from parsed text
│   ├── utilities/             # Base classes and helpers
│   └── synth_pdf/             # Synthetic PDF generation (optional)
├── parsers/                   # Parser implementations
│   ├── pymupdf4llm/
│   ├── llamaparse/
│   ├── mathpix/
│   └── ...                    # Add your own!
├── data/                      # Benchmark datasets
│   ├── 2026-q1-tables-only/  # Table benchmark dataset
│   │   ├── pdfs/
│   │   └── ground_truth/
│   └── 2026-q1-formulas-only/ # Formula benchmark dataset
│       ├── pdfs/
│       └── ground_truth/

Contributions are welcome!

Adding a parser implementation: See Option 2 above for instructions on adding your parser to the repository.

Bug reports and feature requests: Please open an issue on GitHub.

This project is licensed under the MIT License - see the LICENSE file for details.

If you use this benchmark in your research or project, please cite our papers:

@misc{horn2025formulabench,
    title = {Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs},
    author = {Horn, Pius and Keuper, Janis},
    year = {2025},
    eprint={2511.10390},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url = {https://arxiv.org/abs/2512.09874}
}

@misc{horn2026tablebench,
    title = {Benchmarking PDF Parsers on Table Extraction with LLM-based Semantic Evaluation},
    author = {Horn, Pius and Keuper, Janis},
    year = {2026},
    eprint={2603.18652},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url = {https://arxiv.org/abs/2603.18652}
}

📄 Papers: arXiv:2512.09874 (Formulas) | arXiv:2603.18652 (Tables)

This work has been supported by the German Federal Ministry of Research, Technology and Space (BMFTR) in the program "Forschung an Fachhochschulen in Kooperation mit Unternehmen (FH-Kooperativ)" within the joint project LLMpraxis under grant 13FH622KX2.