Skip to content

Add benchmark tracking workflow with GitHub Pages#3028

Merged
dantleech merged 1 commit intophpactor:masterfrom
AJenbo:bench
Mar 9, 2026
Merged

Add benchmark tracking workflow with GitHub Pages#3028
dantleech merged 1 commit intophpactor:masterfrom
AJenbo:bench

Conversation

@AJenbo
Copy link
Contributor

@AJenbo AJenbo commented Mar 7, 2026

This adds a GitHub Actions workflow that tracks PHPBench results over time
using benchmark-action/github-action-benchmark
and publishes them to GitHub Pages.

Here's what it looks like in practice on our project:
https://ajenbo.github.io/phpantom_lsp/dev/bench/

On push to master:

  • Runs the full PHPBench suite
  • Stores results on the gh-pages branch
  • Results are viewable at https://phpactor.github.io/phpactor/dev/bench/

On pull requests:

  • Runs the same benchmarks and compares against the stored baseline
  • Comments on the PR and fails the check if any benchmark regresses beyond 130%

Since PHPBench's XML output isn't directly compatible with the benchmark action,
a small PHP converter script (.github/phpbench_to_json.php) bridges the two formats.

The gh-pages branch is created automatically on the first run.

We use phpactor's benchmarks as the basis for performance testing in PHPantom and wanted to give something back. Hopefully this is useful for tracking regressions upstream too.

@dantleech
Copy link
Collaborator

dantleech commented Mar 7, 2026 •
edited
Loading

nice, this made me wonder if PHPBench could support JSON output by default (PR):

 ./bin/phpbench run examples/Benchmark/ --iterations=2 --report=github-action-benchmark --output=json --progress=none | jq
[
  {
    "name": "HashingBench#benchAlgos(0,0)",
    "unit": "µs",
    "value": "0.8",
    "range": "+0.00%"
  },
  {
    "name": "HashingBench#benchAlgos(1,0)",
    "unit": "µs",
    "value": "0.5",
    "range": "+0.00%"
  },
  {
    "name": "HashingBench#benchAlgos(2,0)",
    "unit": "µs",
    "value": "0.5",
    "range": "+0.00%"
  },
  {
    "name": "HashingBench#benchAlgos(3,0)",
    "unit": "µs",
    "value": "0.7",
    "range": "+0.00%"
  },
  {
    "name": "HashingBench#benchAlgos(4,0)",
    "unit": "µs",
    "value": "0.8",
    "range": "+0.00%"
  },
  {
    "name": "HashingBench#benchAlgos(5,0)",
    "unit": "µs",
    "value": "1.2",
    "range": "-4.00%"
  },
...

Using a currently custom report:

    'github-action-benchmark' => [
        'generator' => 'expression',
        'cols' => [
            'name' => 'format("%s#%s(%s)", first(benchmark_name), first(subject_name), first(variant_name))',
            'unit' => 'coalesce(first(subject_time_unit), "microseconds")',
            'value' => 'display_as_time(first(result_time_avg), first(subject_time_unit))',
            'range' => 'format("%+.2f%%", first(result_comp_deviation))',
        ],
        'aggregate' => ['benchmark_class', 'subject_name', 'variant_index'],
    ],

but maybe I could add it as a default 🤔 or maybe just a documented exeample.

@dantleech dantleech merged commit b5b5adf into phpactor:master Mar 9, 2026
13 checks passed
@dantleech
Copy link
Collaborator

dantleech commented Mar 9, 2026

thanks, I'll try and replace the custom converson script with a PHPBench report 👍

@AJenbo AJenbo deleted the bench branch March 9, 2026 20:49
@dantleech
Copy link
Collaborator

dantleech commented Mar 20, 2026 •
edited
Loading

@AJenbo I feel like I'm being stupd - but the action is failing for 3rd-party PRs: https://github.com/phpactor/phpactor/actions/runs/22880526045/job/66381997235?pr=3031

is that expected?

nm, think we've figured it out: https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#pull_request_target

@dantleech
Copy link
Collaborator

dantleech commented Mar 21, 2026 •
edited
Loading

@AJenbo I've now made the report generation possible natively in the newly tagged PHPBench 1.6.0:

In phpbench.json add this custom report:

{
    // ...
    "report.generators": {
        "github-action-benchmark": {
            "cols": [
                "name",
                "unit",
                "value",
                "range",
                "extra"
            ],
            "expressions": {
                "name": "format('%s::%s%s', first(benchmark_name),first(subject_name),if(first(variant_name) != false, format(' (%s)', first(variant_name)), ''))",
                "unit": "coalesce(time_unit(first(subject_time_unit), true),'µs')",
                "value": "time_convert(mode(result_time_avg), 'us', time_unit(first(subject_time_unit)))",
                "range": "format('± %.2f%%', rstdev(result_time_avg))",
                "extra": "format('%d iterations, %d revs', first(variant_iterations), first(variant_revs))"
            },
            "aggregate": [
                "suite_tag",
                "benchmark_class",
                "subject_name",
                "variant_index"
            ],
            "generator": "expression"
        }
    }
}

You can then generate the JSON report with:

        run: vendor/bin/phpbench run --progress=plain --report=github-action-benchmark --output=json > output.json

@AJenbo
Copy link
Contributor Author

AJenbo commented Mar 22, 2026

Nice!

@llaville llaville mentioned this pull request Mar 23, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AJenbo @dantleech