Implement golden-file matching for `integration_test` on Android and iOS devices by matanlurey · Pull Request #160484 · flutter/flutter

matanlurey · 2024-12-18T02:26:22Z

Work towards #143299.
Work towards #160043.

This PR implements, end-to-end, support for matchesGoldenFile when (a) running with package:integration_test (b) on a device, such as an Android emulator, Android device, iOS simulator, or iOS device, where the runner of a test file does not have process and local-file system access.

There are multiple parts to this PR; I could make it smaller than 1K lines, but the bulk of that is tests, and it would mean landing PRs that are incomplete and unused, which does not seem useful - so instead here is a quick overview of the PR's contents - questions/feedback welcome, and I am willing to break code out or land incremental refactors if requested.

Augmented flutter_platform.dart (used for iOS and Android), similar to flutter_web_platform.dart, now creates and uses test_golden_comparator.dart to proxy calls (coming from the VM service protocol) for golden-file updates and comparisons to a flutter_tester process. A full explanation of how (or why) it works this way is too hard to include here, but see Refactor TestGoldenComparator to be useful for non-web (Android, iOS) integration tests #160215 for more details.
Added VmServiceProxyGoldenFileComparator, which is a currently unused (outside of a single e2e test) comparator that forwards calls to compare and update to the VM service protocol (of which, the other side of this is implemented above, in flutter_platform.dart. The idea is that this comparator would be used automatically when running in an integration test on a device that requires it (similar to how web works today), but that is not wired up yet and requires additional work in flutter_tools.
Added two unit tests (of both the client and server), and a full e2e-test using it to run matchesGoldenFile.

…-integration-test-proxy

andrewkolos

LGTM

yjbanov · 2024-12-26T23:08:46Z

packages/flutter_tools/lib/src/test/flutter_platform.dart

+    final IsolateRef testAppIsolate = await vmService.findExtensionIsolate(_kExtension);
+    await vmService.service.streamListen(_kEventName);
+    vmService.service.onEvent(_kEventName).listen((Event e) async {
+      if (e.extensionKind != 'compare' && e.extensionKind != 'update') {


nit: !['compare', 'update'].contains(e.extensionKind) might read a little better.

yjbanov · 2024-12-26T23:13:19Z

packages/integration_test/lib/src/vm_service_golden_client.dart

+/// <String, Object?>{
+///   'result': true /* or possibly false, in the case of 'compare' calls */
+/// }
+/// ```


Are the details of the message/response public API? Not sure why these need to be under ///. Maybe make them //?

It's not clear to me what is/isn't public API here - for example, another package (flutter_tools) needs to fulfill this API for the feature to work.

I am going to keep it as-is for now.

yjbanov · 2024-12-26T23:15:40Z

packages/integration_test/lib/src/vm_service_golden_client.dart

+    if (kIsWeb) {
+      return false;
+    }
+    return !io.Platform.isAndroid && !io.Platform.isIOS;


Are these the only platforms that separate device from host?

Currently, yes.

yjbanov · 2024-12-26T23:16:42Z

packages/integration_test/lib/src/vm_service_golden_client.dart

+
+  static void _assertNotRunningOnFuchsia() {
+    if (!kIsWeb && io.Platform.isFuchsia) {
+      throw UnsupportedError('Fuchsia is not supported');


Consider a more specific error message, e.g. "integration_test golden testing does not support Fuchsia".

auto-submit · 2024-12-27T22:08:17Z

auto label is removed for flutter/flutter/160484, due to - The status or check suite Merge Queue Guard has failed. Please fix the issues identified (or deflake) before re-applying this label.

auto-submit · 2024-12-27T22:41:22Z

auto label is removed for flutter/flutter/160484, due to - The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label.

…oid and iOS devices (flutter/flutter#160484)

Ignore the golden file associated with packages/integration_test/example/integration_test/integration_test_matches_golden_file.png Test (and functionality) introduced in flutter#160484.

Ignore the golden file associated with packages/integration_test/example/integration_test/integration_test_matches_golden_file.png Test (and functionality) introduced in #160484. Related issues: * #143299. * #160043. ## Pre-launch Checklist - [X] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [X] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [X] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [X] I signed the [CLA]. - [X] I listed at least one issue that this PR fixes in the description above. - [X] I updated/added relevant documentation (doc comments with `///`). - [X] I added new tests to check the change I am making, or this PR is [test-exempt]. - [X] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [X] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord].  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

…oid and iOS devices (flutter/flutter#160484)

matanlurey added 3 commits December 17, 2024 15:36

Add screenshot testing for integration tests.

b30e640

Finish adding tests I think.

a3f8f16

Tweak the test.

76e1071

matanlurey requested review from andrewkolos, camsim99, johnmccutchan and jonahwilliams December 18, 2024 02:26

Merge remote-tracking branch 'upstream/main' into matches-golden-file…

f99ab25

…-integration-test-proxy

matanlurey marked this pull request as ready for review December 19, 2024 21:39

matanlurey added 3 commits December 19, 2024 15:28

Add reduced-test-set.

288aa1a

Merge remote-tracking branch 'upstream/main' into matches-golden-file…

32fa078

…-integration-test-proxy

Format.

60cc030

andrewkolos approved these changes Dec 26, 2024

View reviewed changes

matanlurey added 2 commits December 26, 2024 12:24

++

ae04088

++

b84004b

yjbanov approved these changes Dec 26, 2024

View reviewed changes

matanlurey and others added 2 commits December 27, 2024 12:05

Feedback.

b8a6300

Merge branch 'master' into matches-golden-file-integration-test-proxy

d506514

matanlurey added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 27, 2024

auto-submit bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Dec 27, 2024

matanlurey added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 27, 2024

auto-submit bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Dec 27, 2024

matanlurey added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 27, 2024

auto-submit bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Dec 27, 2024

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 6, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

7ba96c1

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll mentioned this pull request Jan 7, 2025

Roll Flutter from 29a6c648ca4d to cda35151265f (140 revisions) flutter/packages#8380

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 7, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

affeba5

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll mentioned this pull request Jan 7, 2025

Roll Flutter from 29a6c648ca4d to b55290697705 (141 revisions) flutter/packages#8381

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 7, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

d9dc955

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll mentioned this pull request Jan 7, 2025

Roll Flutter from 29a6c648ca4d to e66e04f39ffa (142 revisions) flutter/packages#8382

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 7, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

2ad86de

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll mentioned this pull request Jan 7, 2025

Roll Flutter from 29a6c648ca4d to bb4628e0f046 (143 revisions) flutter/packages#8383

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 7, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

3b7102a

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll mentioned this pull request Jan 7, 2025

Roll Flutter from 29a6c648ca4d to e30bf8523123 (144 revisions) flutter/packages#8385

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 7, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

312b02a

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll mentioned this pull request Jan 7, 2025

Roll Flutter from 29a6c648ca4d to 0cc590860db5 (147 revisions) flutter/packages#8386

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 7, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

9b13569

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll mentioned this pull request Jan 7, 2025

Manual roll Flutter from 29a6c648ca4d to 986fd1436314 (151 revisions) flutter/packages#8387

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 7, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

66d3f05

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 7, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

4512a4a

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 8, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

5f2963f

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 8, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

4ce8813

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 8, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

ecb77ad

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 8, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

84e22e8

…oid and iOS devices (flutter/flutter#160484)

cbracken mentioned this pull request Jan 9, 2025

integration_test: Add gitignore of golden image #161404

Merged

9 tasks

Piinks mentioned this pull request Jan 11, 2025

Flutter Notable Commits - 2025 #161460

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 12, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

ec1bad0

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 13, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

c1700a1

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 13, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

5c62327

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 6, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

4161e8c

…oid and iOS devices (flutter/flutter#160484)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 7, 2025

4cd0e33 Implement golden-file matching for integration_test on Andr…

2daf35c

…oid and iOS devices (flutter/flutter#160484)

victorsanni mentioned this pull request Feb 13, 2026

Add @awaitNotRequired annotation to flutter sdk #181513

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement golden-file matching for `integration_test` on Android and iOS devices#160484

Implement golden-file matching for `integration_test` on Android and iOS devices#160484
auto-submit[bot] merged 11 commits intoflutter:masterfrom
matanlurey:matches-golden-file-integration-test-proxy

matanlurey commented Dec 18, 2024 •

edited

Loading

Uh oh!

andrewkolos left a comment

Uh oh!

yjbanov Dec 26, 2024

Uh oh!

matanlurey Dec 27, 2024

Uh oh!

yjbanov Dec 26, 2024

Uh oh!

matanlurey Dec 27, 2024

Uh oh!

yjbanov Dec 26, 2024

Uh oh!

matanlurey Dec 27, 2024

Uh oh!

yjbanov Dec 26, 2024

Uh oh!

matanlurey Dec 27, 2024

Uh oh!

auto-submit bot commented Dec 27, 2024

Uh oh!

auto-submit bot commented Dec 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

matanlurey commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewkolos left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

auto-submit bot commented Dec 27, 2024

Uh oh!

auto-submit bot commented Dec 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

matanlurey commented Dec 18, 2024 •

edited

Loading