This repository stores benchmark results and datasets collected with metriq-gym. The data here is consumed by metriq-web for presentation and analysis.

Part of the Metriq project.

• Run python3 scripts/aggregate.py (or python3.13 scripts/aggregate.py) to generate aggregated results.
• These scripts use modern Python syntax; use Python >=3.10 (recommended: python3.13).

GitHub Pages publishes the contents of dist/. To preview what will be served at https://unitaryfoundation.github.io/metriq-data/:

python scripts/aggregate.py
cp pages/index.html dist/index.html
python -m http.server --directory dist 8000

Then open http://localhost:8000/.

metriq-score is computed per metric relative to a baseline device, honoring directionality:

• higher-is-better: score = (value / baseline) * 100
• lower-is-better: score = (baseline / value) * 100

Example: Say X is the device baseline for series v0.4. Then for a metric where higher is better (e.g. "fidelity"), we assign a metriq-score of 100 to the value that X scored on that metric. If the raw value of that benchmark on X was 0.5, and another device Y reports 0.9, then the metriq-score of Y is 0.9 / 0.5 * 100 = 180.

Edit scripts/scoring.json, which centralizes both baseline selection and composite scoring.

Example scripts/scoring.json:

{
  "series": {
    "v0.4": {
      "baseline": { "provider": "origin", "device": "origin_wukong" },
      "composite": {
        "components": [
          {
            "label": "BSEQ",
            "weight": "1/2",
            "components": [
              { "benchmark": "BSEQ", "metric": "fraction_connected", "weight": "1/1" }
            ]
          },
          {
            "label": "QML Kernel",
            "weight": "1/2",
            "components": [
              { "benchmark": "QML Kernel", "metric": "accuracy_score", "selector": { "num_qubits": 10 }, "weight": "1/1" }
            ]
          }
        ]
      }
    }
  },
  "default": {
    "baseline": { "provider": "ibm", "device": "ibm_torino" },
    "composite": {
      "components": [
        {
          "label": "BSEQ",
          "weight": "1/2",
          "components": [
            { "benchmark": "BSEQ", "metric": "fraction_connected", "weight": "1/1" }
          ]
        },
        {
          "label": "QML Kernel",
          "weight": "1/2",
          "components": [
            { "benchmark": "QML Kernel", "metric": "accuracy_score", "selector": { "num_qubits": 10 }, "weight": "1/1" }
          ]
        }
      ]
    }
  }
}

Baselines are computed per major series (e.g., all v0.x.y share one baseline reference), using the latest available baseline row per (benchmark, metric, selector) key.

Some of these results used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.