[WIP] Semantics integration test for Android by jonahwilliams · Pull Request #19005 · flutter/flutter

jonahwilliams · 2018-07-02T19:19:25Z

This test still needs more config, and requires talkback to be enabled to function. This can be done through adb:

adb shell settings put secure enabled_accessibility_services com.google.android.marvin.talkback/com.google.android.marvin.talkback.TalkBackService
can be disabled (apparently) with same command replaced with null for the value

Also adds additional finders/matcher for semantics

…tics

…checkbox test

…oid_issues

jonahwilliams · 2018-07-02T22:13:11Z

cc @yjbanov @goderbauer - I'm not quite sure what I need to do to correctly wire this up as an integration test.

This also includes some additions to flutter_test which I think are related to overall testing improvements, but I can split that out into a separate PR if that would be easier to track

goderbauer

+1 for adding integrations tests. We should figure out a similar way for iOS.

Also, can we split this into at least 3 PRs for easier reviewing:

PR for the changes to flutter_test
PR for the changes to flutter_driver
PR for adding the actual integration tests

goderbauer · 2018-07-03T15:53:07Z

dev/integration_tests/semantics/lib/main.dart

+  Widget build(BuildContext context) {
+    return new MaterialApp(
+      routes: <String, WidgetBuilder>{
+        'SelectionControls': (BuildContext context) => new SelectionControlsPage(),


Should these maybe be static consts on SelectionControlPage and TextFieldsPage? e.g. SelectionControlPage.routeName and TextFieldsPage.routeName.

goderbauer · 2018-07-03T15:56:48Z

...ration_tests/semantics/android/app/src/main/java/com/yourcompany/semantics/MainActivity.java

+      result.put("flags", flags);
+      Rect nodeRect = new Rect();
+      node.getBoundsInScreen(nodeRect);
+      rect.put("left", nodeRect.left / mScreenDensity);


Why do we need to correct these values with the density?

This is just so the rects will be in the same space as the rects on the SemanticsData. So you could check that a checkbox is 48 by 48

goderbauer · 2018-07-03T15:57:58Z

dev/integration_tests/semantics/test_driver/android_testing.dart

+  /// A checkbox widget.
+  static const String checkBox = 'android.widget.CheckBox';
+
+  /// The default className if none is provided by flutter.


nit: flutter -> Flutter

goderbauer · 2018-07-03T16:00:17Z

dev/integration_tests/semantics/test_driver/android_testing.dart

+  static const String textView = 'android.widget.TextView';
+}
+
+/// Action constants Taken from [AccessibilityAction].


This is Android's android.view.accessibility.AccessibilityNodeInfo.AccessibilityAction? Maybe make that clearer? Also, I am not sure if dart doc can link to that, so maybe use enclose it in `` instead of []?

goderbauer · 2018-07-03T16:00:57Z

dev/integration_tests/semantics/test_driver/android_testing.dart

+    return id == typedOther.id;
+  }
+
+  /// Taken from [AccessibilityAction.ACTION_FOCUS].


Also here and below.

goderbauer · 2018-07-03T16:16:41Z

packages/flutter/test/material/back_button_test.dart

+  testWidgets('BackButton semantics', (WidgetTester tester) async {
+    final SemanticsHandle handle = tester.ensureSemantics();
+    await tester.pumpWidget(
+        new MaterialApp(


nit: add trailing commas and only indent by two spaces.

Fixed in https://github.com/flutter/flutter/pull/19046/files

goderbauer · 2018-07-03T16:21:34Z

packages/flutter_driver/lib/src/driver/driver.dart

    return result.changedState;
  }

+  Future<int> getSemanticsId(SerializableFinder finder, { Duration timeout = _kShortTimeout}) async {


doc comment?

Addressed in #19047

goderbauer · 2018-07-03T16:22:04Z

packages/flutter_driver/lib/src/common/find.dart

+
+
+/// A Flutter driver command that retrieves a semantics id using a specified
+/// finder.


What happens if the thing identified by finder doesn't own a semantics node?

Addressed in #19047

goderbauer · 2018-07-03T16:22:15Z

packages/flutter_driver/lib/src/common/find.dart

+/// A Flutter driver command that retrieves a semantics id using a specified
+/// finder.
+///
+/// Semantics must be enabled before using this command.


Link to how to enable semantics?

Addressed in #19047

goderbauer · 2018-07-03T16:24:22Z

packages/flutter_test/lib/src/widget_tester.dart

    await tap(backButton);
  }
+
+  /// Attempts to find the [SemanticsData] of first result from [finder].


document what happens if the object identified by finder doesn't own its own semantics node?

addressed in https://github.com/flutter/flutter/pull/19046/files

yjbanov · 2018-07-03T21:11:54Z

can we split this into at least 3 PRs for easier reviewing

I'd even make it 4 PRs: flutter_test, flutter_driver, new test app, driver/devicelab scripts.

Hixie · 2018-08-07T02:16:51Z

It seems a bit... overly complicated that we are sending the semantic data from the framework to the Java code, then back to the framework, then to the driver. Ideally we'd want to check the actual data from Android, as if we were Talkback. Is there some way we can do that instead? Reading the data using the Android accessibility APIs directly?

jonahwilliams · 2018-08-07T16:26:59Z

I don't believe we can use TalkBack directly - in theory we could write our own accessibility service, but there is the added complexity of trying to synchronize three separate processes instead of just two.

Ultimately TalkBack just sees the AccessibilityNodeInfos that I use here - but this at least gives us test coverage of the increasingly complex Android accessibility bridge.

(maybe this is technically an end to end test?)

Hixie · 2018-08-07T22:27:46Z

How do regular android apps test their accessibility?

jonahwilliams · 2018-08-07T22:30:32Z

https://developer.android.com/training/accessibility/testing

A combination of manual testing and a few automated lints like missing labels

jonahwilliams · 2018-08-24T01:07:07Z

Closing in favor of https://github.com/flutter/flutter/compare/master...jonahwilliams:android_a11y_integration?expand=1

…ions) (flutter#19005)" (flutter/engine#19101)

jonahwilliams added 4 commits June 30, 2018 00:01

initial work on android integration testing and new finders for seman…

1d69291

…tics

update semantics data matcher to only require boolean fields, update …

93e9952

…checkbox test

add missing licenses and more semantics test cases, clean up matcher

60da7f9

Merge branch 'master' of https://github.com/flutter/flutter into andr…

c8825f9

…oid_issues

jonahwilliams added the work in progress; do not review label Jul 2, 2018

googlebot added the cla: yes label Jul 2, 2018

goderbauer reviewed Jul 3, 2018

View reviewed changes

Jonah Williams and others added 6 commits July 9, 2018 16:04

merge with upstream

40e25aa

undo accidental merge change

10737e1

undo accidental merge change

b17d12a

merge with upstream

47f8445

more merging

c73f4fb

merge with origin

41aba47

jonahwilliams mentioned this pull request Jul 24, 2018

Create flutter_driver infra for testing the Android AccessibilityNodeInfo generated by Flutter #19700

Merged

jonahwilliams closed this Aug 24, 2018

engine-flutter-autoroll mentioned this pull request Jun 17, 2020

Roll Engine from 965fbbed1776 to e9688f8e362a (106 revisions) #59722

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jun 17, 2020

e9688f8 Revert "Roll Skia from de175abede4d to 32d5cfa1f35e (15 revis…

3009e5a

…ions) (flutter#19005)" (flutter/engine#19101)

engine-flutter-autoroll mentioned this pull request Jun 18, 2020

Roll Engine from 965fbbed1776 to e7ffcb2970f2 (119 revisions) #59776

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jun 18, 2020

e9688f8 Revert "Roll Skia from de175abede4d to 32d5cfa1f35e (15 revis…

25ce4f8

…ions) (flutter#19005)" (flutter/engine#19101)

engine-flutter-autoroll mentioned this pull request Jun 18, 2020

Roll Engine from 965fbbed1776 to 0266e60a0834 (121 revisions) #59783

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jun 18, 2020

e9688f8 Revert "Roll Skia from de175abede4d to 32d5cfa1f35e (15 revis…

c3ecbc4

…ions) (flutter#19005)" (flutter/engine#19101)

engine-flutter-autoroll mentioned this pull request Jun 18, 2020

Roll Engine from 965fbbed1776 to 5157a6b2107d (122 revisions) #59790

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jun 18, 2020

e9688f8 Revert "Roll Skia from de175abede4d to 32d5cfa1f35e (15 revis…

67cfe56

…ions) (flutter#19005)" (flutter/engine#19101)

engine-flutter-autoroll mentioned this pull request Jun 18, 2020

Roll Engine from 965fbbed1776 to b7f8d2d11040 (125 revisions) #59795

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jun 18, 2020

e9688f8 Revert "Roll Skia from de175abede4d to 32d5cfa1f35e (15 revis…

c421da2

…ions) (flutter#19005)" (flutter/engine#19101)

engine-flutter-autoroll mentioned this pull request Jun 18, 2020

Roll Engine from 965fbbed1776 to b5f5e6332cb4 (127 revisions) #59806

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jun 18, 2020

e9688f8 Revert "Roll Skia from de175abede4d to 32d5cfa1f35e (15 revis…

8dc9c72

…ions) (flutter#19005)" (flutter/engine#19101)

engine-flutter-autoroll mentioned this pull request Jun 18, 2020

Roll Engine from 965fbbed1776 to 4a4142c06ffe (130 revisions) #59812

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jun 18, 2020

e9688f8 Revert "Roll Skia from de175abede4d to 32d5cfa1f35e (15 revis…

dc0fbb2

…ions) (flutter#19005)" (flutter/engine#19101)

github-actions bot locked as resolved and limited conversation to collaborators Aug 10, 2021



		/// A Flutter driver command that retrieves a semantics id using a specified
		/// finder.

Conversation

jonahwilliams commented Jul 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonahwilliams commented Jul 2, 2018

Uh oh!

goderbauer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yjbanov commented Jul 3, 2018

Uh oh!

Hixie commented Aug 7, 2018

Uh oh!

jonahwilliams commented Aug 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hixie commented Aug 7, 2018

Uh oh!

jonahwilliams commented Aug 7, 2018

Uh oh!

jonahwilliams commented Aug 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jonahwilliams commented Jul 2, 2018 •

edited

Loading

jonahwilliams commented Aug 7, 2018 •

edited

Loading