With the new integration perf test use cases, this LiveTestWidgetsFlutterBindingFramePolicy.benchmark sounds a little strange. At a first glance, I'd guess that we should use it for perf CI tests. However, it seems to be microbenchmark-only which doesn't wait for the engine VSYNC signal, and wouldn't exercise the engine frame rasterization.

To avoid breaking changes, we probably won't change the name of LiveTestWidgetsFlutterBindingFramePolicy.benchmark. But we may need to document it properly.

Not 100% sure if I understand this benchmark flag but I added some comments in packages/flutter_test/lib/src/binding.dart

We should just document which frame scheduling polcies are incompatible with this method. I think it's probably fine.

Below this: do not throw a string, instead throw a StateError or similar here. I would also avoid documenting too much in the exception message, just say that they're not compatible and document why in the actual method/enum documentation.

@dnfield The exception part is almost a copy of pumpAndSettle. Should I also change that?

If I understand correctly, this clock is a simulated clock that supports time travel (e.g., fast forward or slowdown). So we would render 60 frames for a 1-second-60-fps animation here no matter how slow the actual frame is. That is, if a frame takes 1 second to render, this could spend 1 minute rendering 60 frames and the simulated clock here think it's just 1 second.

Therefore, it may not necessarily simulate and benchmark how an app is performing on a real device. But it would still be very useful in terms of performance testing ( https://mail.google.com/mail/u/0/?pli=1#inbox/FMfcgxwHMZNWmbPBjLTWJhzlnLLfBplZ).

It would be nice if we could have both modes: (1) render only 1 frame for the 1-second-60-fps animation if a frame takes 1 second; (2) render 60 frames for 60 seconds if a frame takes 1 second for the 1-second-60-fps animation.

https://api.flutter.dev/flutter/flutter_test/TestWidgetsFlutterBinding/clock.html I think for such purposes we should override the clock getter in TestWidgetsFlutterBinding, which is what LiveTestWidgetsFlutterBinding and AutomatedTestWidgetsFlutterBinding do:

• LiveTestWidgetsFlutterBinding is using real world clock and therefore gives (1);
• AutomatedTestWidgetsFlutterBinding is using a FakeAsync clock that is controllable. Although I don't find an API to explicitly control it, AutomatedTestWidgetsFlutterBinding.pump forward the clock by the given interval (

  flutter/packages/flutter_test/lib/src/binding.dart

  Line 911 in 59aed97

  _currentFakeAsync.elapse(duration);

  ) and achieves (2).

I'm very unclear as to the value of this method in AutomatedTestWidgetsFlutterBinding. Why aren't we just throwing in that case?

This method seems like it is meant to measure the number of frames pumped for an animation on a real device with a real clock. Why would we want to measure any of that with fakeasync, which could get hung up by any number of problems in a test/benchmark such as trying to do real async work?

To me, it seems the returned frame count isn't very interesting. I'll be more interested in the FrameTiming resulted from the frames driven by this.

From the meeting with Ming just a few minutes ago, I think LiveTestWidgetsFlutterBinding may be less interesting to us in this case as we can just use LiveTestWidgetsFlutterBindingFramePolicy.fullyLive to drive the frames and completely remove the need of pump or pumpContinuous.

The interesting use case of pumpContinuous seems to be implementing a fake clock performance test suggested by Yegor in https://docs.google.com/document/d/1sGh6tvRUUNHI-VtnphliiYNJmmC_jo7afzQtAStPteg/edit?ts=5e80f082#bookmark=id.czojbloii9ww.

I'm very unclear as to the value of this method in AutomatedTestWidgetsFlutterBinding. Why aren't we just throwing in that case?

This method seems like it is meant to measure the number of frames pumped for an animation on a real device with a real clock. Why would we want to measure any of that with fakeasync, which could get hung up by any number of problems in a test/benchmark such as trying to do real async work?

The reason I return an int is to be consistent with pumpAndSettle.

To me, it seems the returned frame count isn't very interesting. I'll be more interested in the FrameTiming resulted from the frames driven by this.

From the meeting with Ming just a few minutes ago, I think LiveTestWidgetsFlutterBinding may be less interesting to us in this case as we can just use LiveTestWidgetsFlutterBindingFramePolicy.fullyLive to drive the frames and completely remove the need of pump or pumpContinuous.

The interesting use case of pumpContinuous seems to be implementing a fake clock performance test suggested by Yegor in https://docs.google.com/document/d/1sGh6tvRUUNHI-VtnphliiYNJmmC_jo7afzQtAStPteg/edit?ts=5e80f082#bookmark=id.czojbloii9ww.

In my case LiveTestWidgetsFlutterBindingFramePolicy.fullyLive does work with an AnimationController-triggered animation, but from the wording of the comments I'm not sure if it's designed for synchronized running of a test app, and if pump is completely not needed (or at least can always be replaced by Future.Delayed). I think I need confirm.

I would find it very helpful to see a test that actually needs this as opposed to just using fullyLive. @liyuqian the linked doc isn't entirely clear to me, but I'm also not as familiar with that metric or its uses. Also, FYI, the doc is google-only.

Just not to be mixed with my other PRs / proposals, this was part of my plan but now it's more like a convenience wrap for infinite animation.

Nit: "This is not for integration performance tests" may be better than "This is not benchmark for general performance test"?

BTW, from the usages of pumpBenchmark, I had an impression that this is for benchmarking the UI thread frame build time only. This does not seem to be suitable for benchmarking the raster thread performance? CC @dnfield @Hixie for confirmation.

This I'm now not sure. We have two places for documenting this:

• https://api.flutter.dev/flutter/flutter_test/LiveTestWidgetsFlutterBindingFramePolicy-class.html
• https://api.flutter.dev/flutter/flutter_test/LiveTestWidgetsFlutterBinding/framePolicy.html
  I'll need to discuss further.

Flutter style says that the first line should be a single sentence.

We should add any further explanations below the first line.

This also is a bit confusing to me, especially given the subsequent documentation below this - maybe we could just amend those lines to explain what types of benchmarks a user is interested in and which types this is meant for?

I'm thinking if we should combine the dartdocs I listed in the above link and change one of them to a link. And it doesn't explicitly say it but I found benchmark flag prevents any ticker triggered animation (which can be inferred from it ignores framework from triggering new frame), which I think should be added to the doc.

Nit: pumpContinuously may be more grammatically correct?

Also CC @dnfield and @Hixie for naming ideas and potential flutter_test API changes.

This sentence isn't quite right - can you explain what you're trying to say here?

@liyuqian #60410 (comment)

My original suggestion was "This is not for integration performance tests". What doesn't sound quite right to me is the "This is not benchmark for" part. I felt that "This is not benchmarking for" may be more correct? If "integration performance tests" sounds strange, how about "Do not use this for integration tests (https://flutter.dev/docs/cookbook/testing/integration)."?

I changed it to " This does not mean benchmarking performance on integration tests."

we should just use 60Hz, which is what we use everywhere else. It doesn't matter what the actual device rate is since this is artificial.

...except maybe you mean this to not be artificial? I admit I don't really understand the use case here.

I used 59.94Hz in pumpFrames first in order to simulate the real devices. @CareF feel free to change pumpFrames if we agree to unify to 60. It shouldn’t break anything since the difference is so small.

we should just use 60Hz, which is what we use everywhere else. It doesn't matter what the actual device rate is since this is artificial.

...except maybe you mean this to not be artificial? I admit I don't really understand the use case here.

I believe there is nothing more than a number. The historical reason I learnt is to avoid resonance with 60Hz city electricity for old TVs and than it is used for newer screens as a convention (but not always respected). I'll change it to 60Hz.

@dkwingsmt It seems that change pumpFrames to 60Hz introduces a Golden file change.

examples should use the snippet mechanism so that it gets syntax highlighting and flutter create templates and so on.

(and most importantly so that it gets analyzed in CI)

Can I have an example how this is done in flutter_test? The only example code in doc string I can find in flutter_test is https://api.flutter.dev/flutter/flutter_test/benchmarkWidgets.html . This example is almost a copy of my unit test.

Can I have an example how this is done in flutter_test? The only example code in doc string I can find in flutter_test is https://api.flutter.dev/flutter/flutter_test/benchmarkWidgets.html . This example is almost a copy of my unit test.

Problem solved.