Skip to content
This repository was archived by the owner on Jun 13, 2024. It is now read-only.

Reenable web benchmarks#604

Merged
ditman merged 2 commits intoflutter:masterfrom
ditman:reenable-web-benchmarks
Feb 23, 2022
Merged

Reenable web benchmarks#604
ditman merged 2 commits intoflutter:masterfrom
ditman:reenable-web-benchmarks

Conversation

@ditman
Copy link
Member

@ditman ditman commented Feb 17, 2022

This PR uses the recently released web_benchmarks: 0.0.7 that supports Chrome 89+ to reenable the web benchmarks here.

There was also a problem in the gallery_automator.dart file, that used dart:io to print output to stdout. dart:io methods cannot be called in a browser, and the benchmarks failed.

I removed the dependency on dart:io from that file, and replaced stdout.writeln by print calls. I had to ignore_for_file:avoid_print because the analyzer thinks that the code I'm touching is "production", when it should be detected as tests.

This is what I get when I run the benchmarks in my machine (looks good, but I need to run each of the tests separately. If I do flutter test test_benchmarks, there seems to be some race condition that makes the benchmarks to never finish (?):

$ flutter test test_benchmarks/web_bundle_size_test.dart 

01:40 +0: Web Compile bundle size
Shell: 
Shell: Building without sound null safety
Shell: For more information see https://dart.dev/null-safety/unsound-null-safety
Shell: 
Shell: Compiling lib/main.dart for the Web...                             89.3s
01:41 +0: Web Compile bundle size
Shell: 4108	/usr/local/google/home/dit/github/flutter_gallery/build/web/main.dart.js
Shell: 1008	/usr/local/google/home/dit/github/flutter_gallery/build/web/main.dart.js.gz
01:41 +1: All tests passed!

----

$ flutter test test_benchmarks/benchmarks_test.dart 

00:02 +0: Can run a web benchmark
Shell: Starting web benchmark tests ...
01:34 +0: Can run a web benchmark                                                                                                      
Launching Chrome.
Launching Google Chrome 98.0.4758.102 

Waiting for the benchmark to report benchmark profile.
[CHROME]: 
[CHROME]: DevTools listening on ws://127.0.0.1:10000/devtools/browser/6514abc1-d1dd-4d74-9e7e-0f2a8c7d9a1d
Connecting to DevTools: ws://localhost:10000/devtools/page/FD8EBB697EF7C84538458956D5AF5609
Connected to Chrome tab:  (http://localhost:9999/index.html)
Launching benchmark "gallery_v2_studies_perf"
[APP] Warming up.
[APP] Warm-up finished.
[APP] ==== List of demos to be run ====
[APP] shrine@study
[APP] rally@study
[APP] crane@study
[APP] fortnightly@study
[APP] reply@study
[APP] starter@study
[APP] ==== End of list of demos to be run ====
[APP] Running demo "shrine@study"
[APP] Running demo "rally@study"
[APP] Running demo "crane@study"
[APP] Running demo "fortnightly@study"
[APP] Running demo "reply@study"
[APP] Running demo "starter@study"
[APP] All demos finished.
Extracted 572 measured frames.
Skipped 123 non-measured frames.
Launching benchmark "gallery_v2_unanimated_perf"
[APP] Warming up.
[APP] Warm-up finished.
[APP] ==== List of demos to be run ====
[APP] app-bar@material
[APP] banner@material
[APP] bottom-app-bar@material
[APP] bottom-navigation@material
[APP] bottom-sheet@material
[APP] button@material
[APP] card@material
[APP] chip@material
[APP] data-table@material
[APP] dialog@material
[APP] divider@material
[APP] grid-lists@material
[APP] lists@material
[APP] menu@material
[APP] nav_drawer@material
[APP] nav_rail@material
[APP] pickers@material
[APP] selection-controls@material
[APP] sliders@material
[APP] snackbars@material
[APP] tabs@material
[APP] text-field@material
[APP] tooltip@material
[APP] cupertino-alerts@cupertino
[APP] cupertino-buttons@cupertino
[APP] cupertino-context-menu@cupertino
[APP] cupertino-navigation-bar@cupertino
[APP] cupertino-picker@cupertino
[APP] cupertino-pull-to-refresh@cupertino
[APP] cupertino-segmented-control@cupertino
[APP] cupertino-slider@cupertino
[APP] cupertino-switch@cupertino
[APP] cupertino-tab-bar@cupertino
[APP] cupertino-text-field@cupertino
[APP] motion@other
[APP] colors@other
[APP] typography@other
[APP] 2d-transformations@other
[APP] ==== End of list of demos to be run ====
[APP] Running demo "app-bar@material"
[APP] Running demo "banner@material"
[APP] Running demo "bottom-app-bar@material"
[APP] Running demo "bottom-navigation@material"
[APP] Running demo "bottom-sheet@material"
[APP] Running demo "button@material"
[APP] Running demo "card@material"
[APP] Running demo "chip@material"
[APP] Running demo "data-table@material"
[APP] Running demo "dialog@material"
[APP] Running demo "divider@material"
[APP] Running demo "grid-lists@material"
[APP] Running demo "lists@material"
[APP] Running demo "menu@material"
[APP] Running demo "nav_drawer@material"
[APP] Running demo "nav_rail@material"
[APP] Running demo "pickers@material"
[APP] Running demo "selection-controls@material"
[APP] Running demo "sliders@material"
[APP] Running demo "snackbars@material"
[APP] Running demo "tabs@material"
[APP] Running demo "text-field@material"
[APP] Running demo "tooltip@material"
[APP] Running demo "cupertino-alerts@cupertino"
[APP] Running demo "cupertino-buttons@cupertino"
[APP] Running demo "cupertino-context-menu@cupertino"
[APP] Running demo "cupertino-navigation-bar@cupertino"
[APP] Running demo "cupertino-picker@cupertino"
[APP] Running demo "cupertino-pull-to-refresh@cupertino"
[APP] Running demo "cupertino-segmented-control@cupertino"
[APP] Running demo "cupertino-slider@cupertino"
[APP] Running demo "cupertino-switch@cupertino"
[APP] Running demo "cupertino-tab-bar@cupertino"
[APP] Running demo "cupertino-text-field@cupertino"
[APP] Running demo "motion@other"
[APP] Running demo "colors@other"
[APP] Running demo "typography@other"
[APP] Running demo "2d-transformations@other"
[APP] All demos finished.
Extracted 3076 measured frames.
Skipped 131 non-measured frames.
Launching benchmark "gallery_v2_animated_perf"
[APP] Warming up.
[APP] Warm-up finished.
[APP] ==== List of demos to be run ====
[APP] progress-indicator@material
[APP] cupertino-activity-indicator@cupertino
[APP] ==== End of list of demos to be run ====
[APP] Running demo "progress-indicator@material"
[APP] Running demo "cupertino-activity-indicator@cupertino"
[APP] All demos finished.
Extracted 849 measured frames.
Skipped 127 non-measured frames.
Launching benchmark "gallery_v2_scroll_perf"
[APP] Warming up.
[APP] Warm-up finished.
[APP] Running scrolling test.
[APP] Scrolling test finished.
Extracted 388 measured frames.
Skipped 132 non-measured frames.
Received profile data
Shell: Web benchmark tests finished.
03:25 +1: All tests passed!

PS: I think this PR should be text-exempt, since it's reenabling a bunch of benchmarks.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read the Flutter Style Guide recently, and have followed its advice.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@ditman
Copy link
Member Author

ditman commented Feb 17, 2022

Tests / Benchmark tests (pull_request) Successful in 8m

All right, this seems to work on the github action! (but it was "flaky" on my machine 😖)

@ditman ditman requested review from guidezpl and pennzht February 17, 2022 02:39
@ditman ditman mentioned this pull request Feb 17, 2022
Closed
@zanderso zanderso mentioned this pull request Feb 17, 2022
Closed
guidezpl
guidezpl approved these changes Feb 22, 2022
View reviewed changes
Copy link
Member

@guidezpl guidezpl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! If this issue pops up again, we may want to avoid concurrent testing of benchmarks in the GitHub Action workflow

@ditman
Copy link
Member Author

ditman commented Feb 23, 2022 •
edited
Loading

Clicking the "Squash and merge" button, @guidezpl feel free to ping me if we need to revisit this!

@ditman ditman merged commit 9164a44 into flutter:master Feb 23, 2022
@ditman ditman deleted the reenable-web-benchmarks branch February 23, 2022 20:22
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@pennzht pennzht Awaiting requested review from pennzht

1 more reviewer

@guidezpl guidezpl guidezpl approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Re-enable web benchmarks

2 participants

@ditman @guidezpl