Make tests more resilient to Skia gold failures and refactor flutter_goldens for extensive technical debt removal by Hixie · Pull Request #140101 · flutter/flutter

Hixie · 2023-12-13T22:32:09Z

Originally landed in #139549
Originally reverted in #140085

Remove all use of global variables.
Always pass in all dependencies, only create them in main or in tests.
Pass in the "print" primitive.
Make all network traffic retry (except when run locally, when it just auto-passes).
Enable tests to be run in random order.
Better error messages

Piinks

LGTM!

…goldens for extensive technical debt removal Originally landed in flutter#139549 Originally reverted in flutter#140085

There were some missing `await`s, one of which in particular caused the tests to not be properly tested on post-submit, which we only discovered because it made Windows fail with access denied errors.

…flutter_goldens for extensive technical debt removal (flutter/flutter#140101)

This is part 4 of a broken down version of the flutter#140101 refactor. This PR renames isAvailableForEnvironment to isRecommendedForEnvironment and replaces a regular expression with a simple function.

This is part 4 of a broken down version of the #140101 refactor. This PR renames isAvailableForEnvironment to isForEnvironment and replaces a regular expression with a simple function. (The latter will change the behaviour for people with branch names like `mainly_refactors` or `chess_master_experiment` or whatever, but I'm pretty sure the old behaviour was not intended.)

…" (#144855) Reverts: #143176 Initiated by: QuncCccccc Reason for reverting: made tree red. Original PR Author: Hixie Reviewed By: {Piinks} This change reverts the following previous change: This is part 4 of a broken down version of the #140101 refactor. This PR renames isAvailableForEnvironment to isForEnvironment and replaces a regular expression with a simple function. (The latter will change the behaviour for people with branch names like `mainly_refactors` or `chess_master_experiment` or whatever, but I'm pretty sure the old behaviour was not intended.)

This is part 4 of a broken down version of the flutter#140101 refactor. This PR renames isAvailableForEnvironment to isForEnvironment and replaces a regular expression with a simple function. (The latter will change the behaviour for people with branch names like `mainly_refactors` or `chess_master_experiment` or whatever, but I'm pretty sure the old behaviour was not intended.)

Hixie · 2024-03-29T21:23:10Z

dev/automated_tests/flutter_test/flutter_gold_expectation.txt

-[0-9]+:[0-9]+ [+]1: Local passes non-existent baseline for new test, empty expectation *
- *No expectations provided by Skia Gold for test: library.flutter.new_golden_test.2.png. This may be a new test. If this is an unexpected result, check https://flutter-gold.skia.org.
- *Validate image output found at flutter/test/library/
 [0-9]+:[0-9]+ [+]2: All tests passed! *


Hixie · 2024-03-29T21:23:55Z

dev/automated_tests/flutter_test/flutter_gold_test.dart

+    ]);
  });
 }



Hixie · 2024-03-29T21:24:07Z

packages/flutter/pubspec.yaml

  vm_service: 13.0.0 # THIS LINE IS AUTOGENERATED - TO UPDATE USE "flutter update-packages --force-upgrade"

-# PUBSPEC CHECKSUM: 84b7
+# PUBSPEC CHECKSUM: e519


Hixie · 2024-03-29T21:24:11Z

packages/flutter/test_private/test/pubspec.yaml

+  typed_data: 1.3.2 # THIS LINE IS AUTOGENERATED - TO UPDATE USE "flutter update-packages --force-upgrade"

-# PUBSPEC CHECKSUM: 942d
+# PUBSPEC CHECKSUM: 5a8e


Hixie · 2024-03-29T21:24:16Z

packages/flutter_goldens/README

+
+ * https://skia.org/docs/dev/testing/skiagold/
+ * https://flutter-gold.skia.org/
+ * https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter


Hixie · 2024-03-29T21:24:22Z

packages/flutter_goldens/dart_test.yaml

-  # This tag tells the test framework to not shuffle the test order according to
-  # the --test-randomize-ordering-seed for the suites that have this tag.
-  no-shuffle:
-    allow_test_randomization: false


Hixie · 2024-04-16T07:32:59Z

packages/flutter_goldens_client/pubspec.yaml

-  # Exclude this package from the hosted API docs.
-  nodoc: true
-
-# PUBSPEC CHECKSUM: 652c


Hixie · 2024-04-16T07:33:53Z

packages/flutter_goldens/test/comparator_selection_test.dart

+    expect(_testRecommendations(os: 'linux', hasCirrus: true, hasGold: true, hasFlutterRoot: true), _Comparator.skip);
+    expect(_testRecommendations(os: 'linux', hasCirrus: true, hasGold: true, hasFlutterRoot: true, hasTryJob: true), _Comparator.skip);
+  });
+}


Hixie · 2024-04-16T07:34:03Z

packages/flutter_goldens/pubspec.yaml

-  path: 1.9.0 # THIS LINE IS AUTOGENERATED - TO UPDATE USE "flutter update-packages --force-upgrade"
  source_span: 1.10.0 # THIS LINE IS AUTOGENERATED - TO UPDATE USE "flutter update-packages --force-upgrade"
  stack_trace: 1.11.1 # THIS LINE IS AUTOGENERATED - TO UPDATE USE "flutter update-packages --force-upgrade"
  stream_channel: 2.1.2 # THIS LINE IS AUTOGENERATED - TO UPDATE USE "flutter update-packages --force-upgrade"


github-actions bot added a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. labels Dec 13, 2023

Hixie force-pushed the skia branch from 6b4e05a to 462d54e Compare December 13, 2023 22:58

Hixie marked this pull request as ready for review December 13, 2023 22:59

Hixie force-pushed the skia branch from 462d54e to d6bd977 Compare December 13, 2023 23:00

Piinks approved these changes Dec 14, 2023

View reviewed changes

Hixie added 2 commits December 21, 2023 12:43

Make tests more resilient to Skia gold failures and refactor flutter_…

e8270de

…goldens for extensive technical debt removal Originally landed in flutter#139549 Originally reverted in flutter#140085

Better error messages and fix race condition

482ff73

There were some missing `await`s, one of which in particular caused the tests to not be properly tested on post-submit, which we only discovered because it made Windows fail with access denied errors.

Hixie force-pushed the skia branch from d6bd977 to 482ff73 Compare December 21, 2023 20:43

Hixie added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 21, 2023

auto-submit bot merged commit 62f1594 into flutter:master Dec 21, 2023

engine-flutter-autoroll mentioned this pull request Dec 22, 2023

Roll Flutter from 11def8eff455 to ede594f8816f (17 revisions) flutter/packages#5739

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 22, 2023

62f1594 Make tests more resilient to Skia gold failures and refactor …

14c4306

…flutter_goldens for extensive technical debt removal (flutter/flutter#140101)

engine-flutter-autoroll mentioned this pull request Dec 22, 2023

Roll Flutter from 11def8eff455 to 48ae5ff6f66d (18 revisions) flutter/packages#5740

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 22, 2023

62f1594 Make tests more resilient to Skia gold failures and refactor …

8c35ea7

…flutter_goldens for extensive technical debt removal (flutter/flutter#140101)

engine-flutter-autoroll mentioned this pull request Dec 22, 2023

Roll Flutter from 11def8eff455 to bba4c1f937d8 (19 revisions) flutter/packages#5741

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 22, 2023

62f1594 Make tests more resilient to Skia gold failures and refactor …

aa652aa

…flutter_goldens for extensive technical debt removal (flutter/flutter#140101)

engine-flutter-autoroll mentioned this pull request Dec 23, 2023

Roll Flutter from 11def8eff455 to cd718db732fe (22 revisions) flutter/packages#5743

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 23, 2023

62f1594 Make tests more resilient to Skia gold failures and refactor …

1cff51f

…flutter_goldens for extensive technical debt removal (flutter/flutter#140101)

engine-flutter-autoroll mentioned this pull request Dec 23, 2023

Roll Flutter from 11def8eff455 to 02d1d64e7f5f (23 revisions) flutter/packages#5744

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 23, 2023

62f1594 Make tests more resilient to Skia gold failures and refactor …

1b0baaf

…flutter_goldens for extensive technical debt removal (flutter/flutter#140101)

engine-flutter-autoroll mentioned this pull request Dec 23, 2023

Roll Flutter from 11def8eff455 to 02d1d64e7f5f (23 revisions) flutter/packages#5745

Closed

engine-flutter-autoroll mentioned this pull request Dec 24, 2023

Roll Flutter from 11def8eff455 to 02d1d64e7f5f (23 revisions) flutter/packages#5746

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 24, 2023

62f1594 Make tests more resilient to Skia gold failures and refactor …

f540b35

…flutter_goldens for extensive technical debt removal (flutter/flutter#140101)

engine-flutter-autoroll mentioned this pull request Dec 26, 2023

Roll Flutter from 11def8eff455 to a86e91f3c451 (24 revisions) flutter/packages#5760

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 26, 2023

62f1594 Make tests more resilient to Skia gold failures and refactor …

1c90663

…flutter_goldens for extensive technical debt removal (flutter/flutter#140101)

engine-flutter-autoroll mentioned this pull request Dec 26, 2023

Roll Flutter from 11def8eff455 to 6c1e11ecc3de (25 revisions) flutter/packages#5761

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 26, 2023

62f1594 Make tests more resilient to Skia gold failures and refactor …

f1aa7a7

…flutter_goldens for extensive technical debt removal (flutter/flutter#140101)

engine-flutter-autoroll mentioned this pull request Dec 26, 2023

Roll Flutter from 11def8eff455 to 6c1e11ecc3de (25 revisions) flutter/packages#5762

Closed

Hixie mentioned this pull request Mar 8, 2024

Factor out use of "print" in flutter_goldens #144846

Merged

auto-submit bot mentioned this pull request Mar 8, 2024

Reverts "Rename isAvailableForEnvironment to isForEnvironment (#143176)" #144855

Merged

Hixie mentioned this pull request Mar 8, 2024

Rename isAvailableForEnvironment to isForEnvironment (#143176) #144858

Merged

This was referenced Mar 17, 2024

Add flutter_goldens README #145278

Merged

Remove state shared across tests #145281

Merged

Hixie commented Mar 29, 2024

View reviewed changes

dev/automated_tests/flutter_test/flutter_gold_test.dart

]);

});

}

Copy link

Contributor Author

Hixie Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

landed

Hixie commented Mar 29, 2024

View reviewed changes

Hixie mentioned this pull request Mar 29, 2024

Make FileSystem dependency explicit througout. #146008

Merged

This was referenced Apr 16, 2024

Assert that the goldenFileComparator is a LocalFileComparator #146802

Merged

Rewrap some lines that are about to get much longer. #146803

Merged

Remove now-redundant tests for isForEnvironment #146804

Merged

Hixie commented Apr 16, 2024

View reviewed changes

Hixie mentioned this pull request Apr 17, 2024

Remove hidden dependencies on the default goldenFileComparator. #146956

Merged

Hixie mentioned this pull request Apr 24, 2024

Remove hidden dependencies on the default LocalPlatform #147342

Merged

This was referenced May 9, 2024

Make FileSystem dependency explicit througout (more). #148095

Merged

Remove hidden dependencies on LocalProcessManager #148096

Merged

This was referenced May 21, 2024

Remove hidden dependencies on HttpClient #148773

Merged

Remove hidden dependency on ABI. #148987

Merged

Hixie mentioned this pull request May 30, 2024

Remove redundant usages of zones in skia_client.dart #149366

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make tests more resilient to Skia gold failures and refactor flutter_goldens for extensive technical debt removal#140101

Make tests more resilient to Skia gold failures and refactor flutter_goldens for extensive technical debt removal#140101
auto-submit[bot] merged 2 commits intoflutter:masterfrom
Hixie:skia

Hixie commented Dec 13, 2023

Uh oh!

Piinks left a comment

Uh oh!

Hixie Mar 29, 2024

Uh oh!

Hixie Mar 29, 2024

Uh oh!

Hixie Mar 29, 2024

Uh oh!

Hixie Mar 29, 2024

Uh oh!

Hixie Mar 29, 2024

Uh oh!

Hixie Mar 29, 2024

Uh oh!

Hixie Apr 16, 2024

Uh oh!

Hixie Apr 16, 2024

Uh oh!

Hixie Apr 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Hixie commented Dec 13, 2023

Uh oh!

Piinks left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants