It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

if it needs to be 15, then it should be restricted to one important test case. Right now this effects 11 tests, so its a large resource increase to the devicelab

All start_up tests seem to be quite noisy to me, and they're of similar importance. I think the best way forward is to reduce the iteration time so we can do a startup iteration in ~10 second so maybe we can do 50 iterations to further reduce the noise without depleting the device lab.

We're never going to be able to do an iteration in 10 seconds

For fun I ran this change on an iPhone 6 with 15 iterations and it reduced complex_layout_ios__start_up from 6:47 to 43 seconds.

Additionally, even if our launch was 99% reliable, running it 50 times would mean that at least one run would fail about 40% of the time.

For fun I ran this change on an iPhone 6 with 15 iterations and it reduced complex_layout_ios__start_up from 6:47 to 43 seconds.

This looks promising. If I understand it correctly, this is 2.87 seconds/iteration which is much faster than my 10 seconds/iteration goal?

Additionally, even if our launch was 99% reliable, running it 50 times would mean that at least one run would fail about 40% of the time.

We could write the benchmark in a way to ignore a certain proportion of failures and only use the successful ones to compute the average. But as a developer, I'd really love to increase the reliability as much as possible (e.g., flutter run should retry certain fragile steps to make the whole command fail less often). The 1% failure may still cost a developer headaches and loss of time.