I added a registry/slow folder for slow tests that will less iterations on presubmit. Waiting for @Hixie's feedback, if we decide to land this we'll have to update the post-submit recipe to run the slow tests as well.

It might be best just to make this a dedicated shard in flutter/flutter rather than using flutter/tests. That would let you just run all the tests, too. I don't think a slow/ registry is a good idea because it'll make people think they can use that too, and then we'll have a bunch of slow tests, and that will quickly get out of control.

I don't believe the problem would be in flutter/flutter where the tests run once, the problem are presubmits (as we run the test 15 times and have a 1 hour limit).

Even if I split the plugins test to it's own Cirrus shard it won't fit within 1 hour if we run it 15 times. Do you see any other way forward other than allowing some tests to be executed less times in presubmits?

If you just make a dedicated flutter/flutter shard (unrelated to flutter/tests) then there's no reason to run the tests 15 times.

I see, your point is that if it flakes in post-submit it will be easier to root cause. Right?

We'll still need a way to specify to not run it 15 times in flutter/tests presubmits though

Sorry I'm not conveying what I mean clearly! I'm suggesting that the plugins tests have nothing to do with flutter/tests or customer_testing at all. We'd be creating a dedicated shard in flutter/flutter, that directly checks out the plugins tests and runs them. There'd be no flutter/tests presubmits to worry about because flutter/tests wouldn't be involved in any way.

Makes sense, thanks for the clarification.

We'd be creating a dedicated shard in flutter/flutter, that directly checks out the plugins tests and runs them. There'd be no flutter/tests presubmits to worry about because flutter/tests wouldn't be involved in any way.

@Hixie As this landed in flutter/flutter#70887, it clones HEAD, which means builds are no longer exactly reproducible, and the release branches will probably break. I think at the minimum the revision should be pinned, customer_testing makes sense as it's an existing framework that we use for declaring external repo dependencies and pinning them.

See flutter/flutter#70887 (comment) for a way to fix that.
The problem with using customer_testing is that it's intended for third-party tests that run in a very short time. It's important that we present a simple mechanism for people to contribute their tests; the more subtle we make it, the harder it will be for people to understand our policies and so forth. These tests just don't work in that framework.

The problem with using customer_testing is that it's intended for third-party tests that run in a very short time.

Would we accept a third party plugin tests that ran in a short time to customer_testing? Is there anything that special about 95% popular 1p battery vs 99% popular 3rd party geolocator (randomly selected). Related to discussion about 1p plugins and the capacity of the ecosystems team. Maybe we need to make that simple mechanism for plugin authors to contribute their tests, and try it out on our 1p plugins.

We would absolutely accept such tests. The simple mechanism is described in https://github.com/flutter/tests/blob/master/README.md and https://github.com/flutter/tests/blob/master/registry/template.test.