Skip to content

Support failures-only and silent reporters in flutter test#148739

Merged
auto-submit[bot] merged 1 commit intoflutter:masterfrom
gnprice:pr-test-failures-only
Jun 3, 2024
Merged

Support failures-only and silent reporters in flutter test#148739
auto-submit[bot] merged 1 commit intoflutter:masterfrom
gnprice:pr-test-failures-only

Conversation

@gnprice
Copy link
Member

@gnprice gnprice commented May 21, 2024

Fixes #148738.

Also add some tests for this part of the flutter test CLI.

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@github-actions github-actions bot added the tool Affects the "flutter" command-line tool. See also t: labels. label May 21, 2024
@gnprice gnprice mentioned this pull request May 21, 2024
Closed
@gnprice
Support failures-only and silent reporters in flutter test
1ede975 
Fixes flutter#148738.

Also add some tests for this part of the `flutter test` CLI.
@gnprice gnprice force-pushed the pr-test-failures-only branch from 218830e to 1ede975 Compare May 21, 2024 16:57
@gnprice
Copy link
Member Author

gnprice commented May 21, 2024

Test flake in CI; rebased to pick up #148737.

@andrewkolos andrewkolos self-requested a review May 30, 2024 20:03
@andrewkolos
Copy link
Contributor

andrewkolos commented May 30, 2024

Note for self: when this lands, we should consider using failures-only for flutter tool CI tests that use package:test

andrewkolos
andrewkolos approved these changes Jun 3, 2024
View reviewed changes
Copy link
Contributor

@andrewkolos andrewkolos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@andrewkolos andrewkolos added the autosubmit Merge PR when tree becomes green via auto submit App label Jun 3, 2024
@auto-submit auto-submit bot merged commit 5e448f4 into flutter:master Jun 3, 2024
@andrewkolos andrewkolos mentioned this pull request Jun 3, 2024
Closed
@andrewkolos
Copy link
Contributor

andrewkolos commented Jun 3, 2024

Note for self: when this lands, we should consider using failures-only for flutter tool CI tests that use package:test

Filed #149611 to track this.

@gnprice gnprice deleted the pr-test-failures-only branch June 3, 2024 20:04
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request Jun 3, 2024
Closed
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jun 3, 2024
@engine-flutter-autoroll
5e448f4 Support failures-only and silent reporters in flutter test (f…
9695362 
…lutter/flutter#148739)
hello-coder-xu pushed a commit to hello-coder-xu/flutter that referenced this pull request Jun 4, 2024
Merge branch 'master' into fix/_floatingActionButtonVisibilityValue-u…
151f277 
…pdate

* master: (168 commits)
  Fix the scrolling layout deviation of `CupertinoActionSheet` (flutter#149439)
  Roll Flutter Engine from 60a7bb2 to a6aa5d8 (2 revisions) (flutter#149627)
  Roll Flutter Engine from ea72558 to 60a7bb2 (2 revisions) (flutter#149623)
  Place `flutter_gpu` in the package cache. (flutter#149299)
  Switch to triage-* labels for platform package triage (flutter#149614)
  Roll pub packages (flutter#149617)
  Fixes multi line textfield hint text gets ellipsized (flutter#148423)
  Support failures-only and silent reporters in `flutter test` (flutter#148739)
  [CupertinoActionSheet] Fix overflow of the overscroll section when the user scrolls far (flutter#149542)
  Fix InputDecorator.prefixIcon color when disabled (flutter#149595)
  Added filter callback on dropdown menu (flutter#143939)
  update generated localized message files in the stocks test app (flutter#148741)
  Add a simplified SimpleCascadingMenuApp example (flutter#149147)
  Reland "Prevent LayoutBuilder from rebuilding more than once (flutter#147856)" (flutter#149303)
  Move some benchmarks from MotoG4 to Mokey (flutter#149567)
  Roll Packages from d8e8e8c to 11e192a (2 revisions) (flutter#149596)
  Cleanup triage reports from docs/ (flutter#149545)
  Roll Flutter Engine from d81edf6 to ea72558 (1 revision) (flutter#149590)
  Roll Flutter Engine from b0f4d74 to d81edf6 (1 revision) (flutter#149468)
  Roll Flutter Engine from 40b868e to b0f4d74 (1 revision) (flutter#149467)
  ...
hello-coder-xu added a commit to hello-coder-xu/flutter that referenced this pull request Jun 6, 2024
@hello-coder-xu
Merge branch 'master' into fix/_floatingActionButtonVisibilityValue-u…
3846fea 
…pdate

* master: (168 commits)
  Fix the scrolling layout deviation of `CupertinoActionSheet` (flutter#149439)
  Roll Flutter Engine from 60a7bb2 to a6aa5d8 (2 revisions) (flutter#149627)
  Roll Flutter Engine from ea72558 to 60a7bb2 (2 revisions) (flutter#149623)
  Place `flutter_gpu` in the package cache. (flutter#149299)
  Switch to triage-* labels for platform package triage (flutter#149614)
  Roll pub packages (flutter#149617)
  Fixes multi line textfield hint text gets ellipsized (flutter#148423)
  Support failures-only and silent reporters in `flutter test` (flutter#148739)
  [CupertinoActionSheet] Fix overflow of the overscroll section when the user scrolls far (flutter#149542)
  Fix InputDecorator.prefixIcon color when disabled (flutter#149595)
  Added filter callback on dropdown menu (flutter#143939)
  update generated localized message files in the stocks test app (flutter#148741)
  Add a simplified SimpleCascadingMenuApp example (flutter#149147)
  Reland "Prevent LayoutBuilder from rebuilding more than once (flutter#147856)" (flutter#149303)
  Move some benchmarks from MotoG4 to Mokey (flutter#149567)
  Roll Packages from d8e8e8c to 11e192a (2 revisions) (flutter#149596)
  Cleanup triage reports from docs/ (flutter#149545)
  Roll Flutter Engine from d81edf6 to ea72558 (1 revision) (flutter#149590)
  Roll Flutter Engine from b0f4d74 to d81edf6 (1 revision) (flutter#149468)
  Roll Flutter Engine from 40b868e to b0f4d74 (1 revision) (flutter#149467)
  ...
hello-coder-xu added a commit to hello-coder-xu/flutter that referenced this pull request Jun 9, 2024
@hello-coder-xu
Merge branch 'master' into fix/_floatingActionButtonVisibilityValue-u…
26adfe6 
…pdate

* master: (181 commits)
  Fix the scrolling layout deviation of `CupertinoActionSheet` (flutter#149439)
  Roll Flutter Engine from 60a7bb2 to a6aa5d8 (2 revisions) (flutter#149627)
  Roll Flutter Engine from ea72558 to 60a7bb2 (2 revisions) (flutter#149623)
  Place `flutter_gpu` in the package cache. (flutter#149299)
  Switch to triage-* labels for platform package triage (flutter#149614)
  Roll pub packages (flutter#149617)
  Fixes multi line textfield hint text gets ellipsized (flutter#148423)
  Support failures-only and silent reporters in `flutter test` (flutter#148739)
  [CupertinoActionSheet] Fix overflow of the overscroll section when the user scrolls far (flutter#149542)
  Fix InputDecorator.prefixIcon color when disabled (flutter#149595)
  Added filter callback on dropdown menu (flutter#143939)
  update generated localized message files in the stocks test app (flutter#148741)
  Add a simplified SimpleCascadingMenuApp example (flutter#149147)
  Reland "Prevent LayoutBuilder from rebuilding more than once (flutter#147856)" (flutter#149303)
  Move some benchmarks from MotoG4 to Mokey (flutter#149567)
  Roll Packages from d8e8e8c to 11e192a (2 revisions) (flutter#149596)
  Cleanup triage reports from docs/ (flutter#149545)
  Roll Flutter Engine from d81edf6 to ea72558 (1 revision) (flutter#149590)
  Roll Flutter Engine from b0f4d74 to d81edf6 (1 revision) (flutter#149468)
  Roll Flutter Engine from 40b868e to b0f4d74 (1 revision) (flutter#149467)
  ...
hello-coder-xu added a commit to hello-coder-xu/flutter that referenced this pull request Jun 11, 2024
@hello-coder-xu
Merge branch 'master' into fix/pageController_get_value_error
3036b33 
* master: (181 commits)
  Fix the scrolling layout deviation of `CupertinoActionSheet` (flutter#149439)
  Roll Flutter Engine from 60a7bb2 to a6aa5d8 (2 revisions) (flutter#149627)
  Roll Flutter Engine from ea72558 to 60a7bb2 (2 revisions) (flutter#149623)
  Place `flutter_gpu` in the package cache. (flutter#149299)
  Switch to triage-* labels for platform package triage (flutter#149614)
  Roll pub packages (flutter#149617)
  Fixes multi line textfield hint text gets ellipsized (flutter#148423)
  Support failures-only and silent reporters in `flutter test` (flutter#148739)
  [CupertinoActionSheet] Fix overflow of the overscroll section when the user scrolls far (flutter#149542)
  Fix InputDecorator.prefixIcon color when disabled (flutter#149595)
  Added filter callback on dropdown menu (flutter#143939)
  update generated localized message files in the stocks test app (flutter#148741)
  Add a simplified SimpleCascadingMenuApp example (flutter#149147)
  Reland "Prevent LayoutBuilder from rebuilding more than once (flutter#147856)" (flutter#149303)
  Move some benchmarks from MotoG4 to Mokey (flutter#149567)
  Roll Packages from d8e8e8c to 11e192a (2 revisions) (flutter#149596)
  Cleanup triage reports from docs/ (flutter#149545)
  Roll Flutter Engine from d81edf6 to ea72558 (1 revision) (flutter#149590)
  Roll Flutter Engine from b0f4d74 to d81edf6 (1 revision) (flutter#149468)
  Roll Flutter Engine from 40b868e to b0f4d74 (1 revision) (flutter#149467)
  ...
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Aug 6, 2024
@engine-flutter-autoroll
5e448f4 Support failures-only and silent reporters in flutter test (f…
179987e 
…lutter/flutter#148739)
gnprice added a commit to gnprice/zulip-flutter that referenced this pull request Feb 13, 2026
@gnprice
check: In CI, show only failing tests, not passing
ce2994d 
Currently the output of our `test` suite, running `flutter test`,
ends with:

  🎉 3502 tests passed, 9 skipped.

That's preceded by about 3511 lines of test output, one for each
test case.  That's a lot of output, and the GitHub Actions log
viewer can be pretty slow to get through it.

In particular, when a test breaks somewhere in this suite in CI, it
can be rather a pain to get to the details of the failure.  It's
there in the log, but somewhere amidst 3500+ lines of other output.

Instead, in CI, tell `flutter test` to skip printing anything for
all the passing and skipped tests.  This is a feature that the Dart
test runner grew a couple of years ago, for exactly this purpose:
  flutter/flutter#27845
  dart-lang/test#829
  flutter/flutter#148739

Don't have the check script use this flag by default, though,
because it's usually not desirable for interactive use in a
terminal.  There, the test runner already makes good use of the
medium: it constantly replaces a single line of output as it
progresses through all the non-failing tests.
@gnprice gnprice mentioned this pull request Feb 13, 2026
Merged
gnprice added a commit to gnprice/zulip-flutter that referenced this pull request Feb 13, 2026
@gnprice
check: In CI, show only failing tests, not passing
eab7855 
Currently the output of our `test` suite, running `flutter test`,
ends with:

  🎉 3502 tests passed, 9 skipped.

That's preceded by about 3511 lines of test output, one for each
test case.  That's a lot of output, and the GitHub Actions log
viewer can be pretty slow to get through it.

In particular, when a test breaks somewhere in this suite in CI, it
can be rather a pain to get to the details of the failure.  It's
there in the log, but somewhere amidst 3500+ lines of other output.

Instead, in CI, tell `flutter test` to skip printing anything for
all the passing and skipped tests.  This is a feature that the Dart
test runner grew a couple of years ago, for exactly this purpose:
  flutter/flutter#27845
  dart-lang/test#829
  flutter/flutter#148739

Don't have the check script use this flag by default, though,
because it's usually not desirable for interactive use in a
terminal.  There, the test runner already makes good use of the
medium: it constantly replaces a single line of output as it
progresses through all the non-failing tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkolos andrewkolos andrewkolos approved these changes

Assignees

No one assigned

Labels

autosubmit Merge PR when tree becomes green via auto submit App tool Affects the "flutter" command-line tool. See also t: labels.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Failures-only reporter for flutter test

2 participants

@gnprice @andrewkolos