Remove no-shuffle from framework tests, part 1: easy cases#123751
Remove no-shuffle from framework tests, part 1: easy cases#123751auto-submit[bot] merged 5 commits intoflutter:masterfrom
Conversation
goderbauer
left a comment
There was a problem hiding this comment.
Thank you sooo much!
LGTM! LGTM! LGTM!
|
Thanks!
Because for some of these test files I couldn't reproduce the problem and don't know what fixed it, I was a bit concerned that one or more of them might just have a low-frequency flake I didn't happen to hit. So I did additional test runs overnight: 500 of them, for a total of 800, all of which passed. The largest number of tests any of these files have is 104, so if any of them were to have an ordering-dependent flake it's unlikely it would be at a rate lower than 1/104 = 0.96%. (That'd be the probability of a given test coming first, or of coming immediately after another given test.) Running on 800 seeds is enough data to reject that at p<.0004. |
Thanks for verifying that. Even if there still is an interdependency or two left in there, it's at a low enough likelihood that we can just deal with it if/when it comes up, which would also expose a seed at which it happens that would let us debug it. |
…23751) Remove no-shuffle from framework tests, part 1: easy cases

Fixes part of #85160.
I have a draft branch which removes
no-shuffletags from all tests in the framework.This PR collects up the "easy" fixes:
tearDown.For all of these files, I reran the tests after my changes both with the previously failing seed mentioned in the comment, and with 300 random seeds, and they passed every time. That's enough to detect a 1% flake rate with p<.05, and a 2% flake rate with p<.003.
The fixes I've left for separate PRs are those where the fix is more complex, and those where I believe changes in the
flutter_testpackage are appropriate in order to make it easier either to diagnose the problem or to avoid it in the first place.Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.