Benchmark suite for evaluating how AI coding agents leverage external context retrieval tools on realistic developer tasks in large, enterprise-scale codebases.

This repository contains:

• 275 benchmark tasks across 20 suites covering 9 developer work types
• Versioned benchmark suites with dual-verifier support (benchmarks/suites/)
• Auditable results snapshots with per-task traces, scores, timing, and cost (runs/snapshots/)
• Metrics extraction and reporting pipelines for score/cost/retrieval analysis
• Task indexes by complexity, language, repo size, and multi-repo scope (benchmarks/indexes/)

Tasks are executed via the Harbor runner with the Claude Code agent harness.

Official results are available as frozen, auditable snapshots in runs/snapshots/.

Each snapshot includes:

• browse.html — Interactive results browser (open in any web browser)
• SNAPSHOT.json — Machine-readable manifest
• export/traces/ — Per-task result metadata, reward scores, agent instructions
• export/summary/ — Aggregate scores, timing, cost breakdowns

Full agent trajectories are available as compressed archives in the corresponding GitHub Release.

Suite definitions in benchmarks/suites/ specify exactly which tasks are in each benchmark version:

Task indexes in benchmarks/indexes/ provide views by:

• Complexity — scored by repo size, repo count, GT files, difficulty
• Language — 13 languages (Go, C++, Java, Python, Rust, C#, JS, TS, ...)
• Repo size — small (<100MB) through xlarge (>5GB)
• Multi-repo — 136 multi-repo tasks across 28 repo sets

All tasks represent realistic developer work in large, often multi-repo, enterprise codebases. Tasks are organized by developer work type.

Tasks are evaluated across multiple MCP configurations to measure the impact of external code retrieval tools:

benchmarks/
  suites/                # Versioned benchmark suite definitions (JSON)
  indexes/               # Task indexes by complexity, language, repo size, multi-repo
  tasks/                 # 275 task definitions across 20 suite directories
    csb_sdlc_feature/    #   Feature implementation (23 tasks)
    csb_sdlc_fix/        #   Bug repair (19 tasks)
    csb_org_migration/   #   Framework migration (25 tasks)
    ...                  #   (20 suites total)
runs/
  snapshots/             # Frozen, auditable result snapshots
    {snapshot_id}/
      SNAPSHOT.json      #   Manifest: suite, model, configs, aggregates
      browse.html        #   Interactive results browser
      export/            #   Sanitized traces for public consumption
scripts/                 # Metrics extraction, evaluation, and operational tooling
  publishing/            #   Snapshot creation, verification, export
  evaluation/            #   IR metrics, retrieval analysis, judge pipelines
  csb_metrics/           #   Python package: models, extractors, discovery
docs/
  technical_reports/     # Published technical reports
  official_results/      # Pointer to runs/snapshots/

git clone https://github.com/sourcegraph/CodeScaleBench.git
cd CodeScaleBench

# Browse benchmark suites
cat benchmarks/suites/csb-v2-dual264.json | python3 -m json.tool | head -20

# Explore task indexes
cat benchmarks/indexes/by-complexity.json | python3 -m json.tool | head -30

# Open results browser
open runs/snapshots/csb-v1-mixed371--haiku45--030326/browse.html

# Pre-flight checks
python3 scripts/infra/check_infra.py

# Run canonical suite
./configs/harnesses/run_selected_tasks.sh \
  --suite-file benchmarks/suites/csb-v2-dual264.json

CodeScaleBench includes a multi-stage QA pipeline:

See LICENSE.