Use case

For device lab tests, if we run A/B test, we can pass in --ab=10, which runs the test 10X2 times and accumulates the results.

Sometimes I just want to compare the performance of several local engine branches (e.g. local main, branch1, branch2, branch3), I don't want to know how they perform compared to non-local engine (which is the A/B test)

Proposal

Option 1:
Upgrade the A/B tests to allow a flag that takes a list of local branch names. e.g. instead of doing --ab=10, we do --count=10 branches=main,branch1,branch2,branch3...

Option 2:
Just allow non-A/B tests to run multiple times and accumulate the result. (e.g. --count=10). It should accumulate the result, and I will compare different branches manually.