Delegate a11y events and action to/from embedded Android platform views.#8250
Delegate a11y events and action to/from embedded Android platform views.#8250amirh merged 6 commits intoflutter:masterfrom
Conversation
This handles delegation of: * AccessibilityNodeProvider#performAction * ViewGroup#requestSendAccessibilityEvent * View#onHoverEvent Additionaly updates the curretly accessibility focused node state that is tracked by the a11y bridge when an embedded view's node is focused.
| * | ||
| * This is used to propagate an updatable reference to the accessibility bridge within the package. | ||
| */ | ||
| class CurrentAccessibilityBridge { |
There was a problem hiding this comment.
Can you describe why we need this? It looks like it's enabling switching out an instance without letting its client know that the value switched. But I'm not sure why that's a good thing to do instead of just offering a mutator on the client and avoiding this class entirely.
There was a problem hiding this comment.
In this case the client doesn't need to know that a change happened, it just needs to know the current value when it needs to use it. Note that this is not public API but a package private mechanism.
I'm assuming that by "offering a mutator" you mean exposing a setAccessibilityBridge call(?). I tend to prefer this solution where there's a single source of truth for the current a11y bridge vs. keeping N references in N clients, and propagating N set calls multiple levels down every time the value is changed, whether that new value is needed or not (in most common cases where a11y is off the value will never be used).
I agree that it's ugly to have a dedicated class just for this but I like the alternatives we've considered less (propagating set calls, or making this class expose update callbacks).
Happy to be convinced otherwise if you have good reasons to prefer a different approach.
There was a problem hiding this comment.
In terms of this chosen approach, I think the only recommendation I would have is that you consider calling this class AccessibilityBridgeHolder because the concept of something being "current" is actually a runtime statement, and ending with AccessibilityBridge makes this class sound like a subclass of AccessibilityBridge.
With regard to this approach vs others, you suggested that there might be many setter methods required to replace this class, and that many would never be used. I'm curious why we would add setters, or even expose this class, to things that won't need it? I'm also curious how this approach reduces exposure across the API surface. Considering all of those classes that would have setters added to them, aren't they all being given a reference to this class, anyway?
If your analysis says that this is the most readable and maintainable option of those that exist, then maybe I'm just missing something. But I would definitely recommend considering if this approach isn't simply re-inventing what we would normally do with a variable and a setter. If it is, then it will probably be less readable and less maintainable over time for developers who are used to simply setting references on things. But it's up to you.
There was a problem hiding this comment.
@amirh (optional) I thought of another potential alternative besides this, having to propagate all the set calls, or having to register callbacks. You could get rid of the getCurrent method on this holder class and instead wrap and expose all the AccessibilityBridge methods that you're expecting clients to call, so none of the clients are ever dealing with the Bridge directly and able to cache it. It would mean extra boilerplate and a maintenance burden though.
@matthew-carroll I think the root issue here is that there's a convoluted and long chain of instantiations, some of which require dependencies from upper levels. The same kind of issue that dependency injection practices and frameworks try to prevent. We need AccessiblityBridge constructed and maintained in PlatformViewsController because of onChange callbacks, but then it's used way over in AccessibilityDelegatingFrameLayout, which is constructed through an internal chain of new calls. So if setters were used PlatformViewsController would need to call VirtualDisplayController#setBridge which would need to call SingleViewPresentation#setBridge which would need to call AccessibilityDelegatingFrameLayout#setBridge which would then be the final thing to actually use the bridge. Pub/sub systems (like EventBus also address this type of problem, but if we want to bring in something like either that or use DI here it would be its own discussion/broader refactor.
There was a problem hiding this comment.
Went with @mklim's suggestion, made this be an AccessibilityEventsDelegate.
We could potentially take this a step further in making sure no-one but PlatformViewsController sets the a11y bridge by splitting this into an interface and an implementation. But I think that this extra complexity isn't justified, as this is an internal class.
| // | ||
| // This is the ID of a node generated by the AccessibilityViewEmbedder if an embedded node is focused, | ||
| // -1 otherwise. | ||
| private int embeddedFocusedNodeId; |
There was a problem hiding this comment.
Can we just make this an Integer and use null for non-existence rather than a magic number?
| } | ||
| } | ||
|
|
||
| public boolean externalViewRequestSendAccessibilityEvent(View embeddedView, View eventOrigin, AccessibilityEvent event) { |
There was a problem hiding this comment.
Can you add appropriate javadocs to this method?
|
|
||
| /** | ||
| * Delegates an AccessibilityNodeProvider#requestSendAccessibilityEvent from the AccessibilityBridge to the embedded | ||
| * view. |
There was a problem hiding this comment.
Can you include the meaning of the return value in the javadoc?
|
|
||
| /** | ||
| * Delegates an AccessibilityNodeProvider#performAction from the AccessibilityBridge to the embedded view's | ||
| * accessibility node provider. |
There was a problem hiding this comment.
Same here with the return value.
| return provider.performAction(origin.id, accessibilityAction, arguments); | ||
| } | ||
|
|
||
| public Integer getRecordFlutterId(View embeddedView, AccessibilityRecord event) { |
There was a problem hiding this comment.
Can you add a javadoc to this method?
|
I added @mklim for Java readability. He has been providing readability review on all of my embedding PRs and it would probably be a good idea to get similar input for this series of PRs as well. |
| class CurrentAccessibilityBridge { | ||
| private AccessibilityBridge accessibilityBridge; | ||
|
|
||
| public AccessibilityBridge getCurrentAccessibilityBridge() { |
There was a problem hiding this comment.
I think this could use some huge warning labels on it for how long "current" is valid for.
| return accessibilityBridge; | ||
| } | ||
|
|
||
| public void setCurrentAccessibilityBridge(AccessibilityBridge accessibilityBridge) { |
There was a problem hiding this comment.
I'm not sure the right way tor restrict this, but there's a potential bug here since clients could call this and override the actual current bridge with whatever they've cached from getCurrent. In general exposing this just makes it kind of unclear for how this state should be managed. I'd add least add code comments saying that this should only ever be called by the class controlling the AccessibilityBridge.
Would it be potentially worth taking lambda or something similar in the constructor and updating the bridge internally when it's triggered? That way the only class that can mess with the state is the person that constructed this originally. Not sure if that code complexity would be worth the guarantee of really guarding this setter though. I think it would end up being extremely verbose. Since we're just using this internally I think this is fine as long as it has a comment on it saying when this should be called.
There was a problem hiding this comment.
So this suggestion is to do something like:
class AccessibilityEventsDelegate {
private AccessibilityBridge accessibilityBridge;
AccessibilityEventsDelegate(OnAccessibilityBridgeUpdated onAccessibilityBridgeUpdated) {
onAccessibilityBridgeUpdated.setListener((accessibilityBridge) -> this.accessibilityBridge = accessibilityBridge );
}
}And then have PlatformViewsController to pass in a callback that's updated on attach/detach.
As noted above I don't believe the use circumstances here justifies this kind of code complexity.
| * | ||
| * This is used to propagate an updatable reference to the accessibility bridge within the package. | ||
| */ | ||
| class CurrentAccessibilityBridge { |
There was a problem hiding this comment.
@amirh (optional) I thought of another potential alternative besides this, having to propagate all the set calls, or having to register callbacks. You could get rid of the getCurrent method on this holder class and instead wrap and expose all the AccessibilityBridge methods that you're expecting clients to call, so none of the clients are ever dealing with the Bridge directly and able to cache it. It would mean extra boilerplate and a maintenance burden though.
@matthew-carroll I think the root issue here is that there's a convoluted and long chain of instantiations, some of which require dependencies from upper levels. The same kind of issue that dependency injection practices and frameworks try to prevent. We need AccessiblityBridge constructed and maintained in PlatformViewsController because of onChange callbacks, but then it's used way over in AccessibilityDelegatingFrameLayout, which is constructed through an internal chain of new calls. So if setters were used PlatformViewsController would need to call VirtualDisplayController#setBridge which would need to call SingleViewPresentation#setBridge which would need to call AccessibilityDelegatingFrameLayout#setBridge which would then be the final thing to actually use the bridge. Pub/sub systems (like EventBus also address this type of problem, but if we want to bring in something like either that or use DI here it would be its own discussion/broader refactor.
|
|
||
| // The accessibility bridge to which accessibility events form the platform views will be dispatched. | ||
| private AccessibilityBridge accessibilityBridge; | ||
| private final CurrentAccessibilityBridge mCurrentAccessibilityBridge; |
There was a problem hiding this comment.
nit: Please rename to just currentAccessibilityBridge for this and the related variables.
https://google.github.io/styleguide/javaguide.html#s5.1-identifier-names
There was a problem hiding this comment.
Our codebase right now is in this weird state were part of the newer files are following the Google Style guide and files from up to around 8 months ago are following something similar to the Android style guide.
We should migrate to a single style, but I think it should be done in dedicated PRs, and I wouldn't want to introduce mixed style to this file in this PR. I think we should follow up with a separate PR to rename all the fields here.
| } | ||
| int recordOriginVirtualID = ReflectionAccessors.getVirtualNodeId(recordOriginPackedId); | ||
| Integer recordFlutterId = originToFlutterId.get(new ViewAndId(embeddedView, recordOriginVirtualID)); | ||
| if (recordFlutterId == null) { |
There was a problem hiding this comment.
(subjective, not in scope for this change, doesn't need to be resolved): I think relying on null as a silent error condition like here is dangerous. It's hard to know when to look for it. Especially on a type like Integer that tends to get equated with the non nullable int. I'd rather there was a constant for INVALID that callers were expected to check for, and the method returned an int instead.
There was a problem hiding this comment.
FYI, I actually asked @amirh to switch from a magic number to null. Can you elaborate on why null would be bad? Null is often used to express non-existence, whereas a value definitely exists but has some special meaning that is different than all other numbers. To me, the latter seemed more likely to confuse people, especially if you ever do a debug dump and see just the magic number instead of null. Are there likely situations where null would create issues?
There was a problem hiding this comment.
Changed to check containsKey's return value.
There was a problem hiding this comment.
Sure thing. I do think this is subjective so the patch as it is is also fine. And I agree magic numbers are unclear, so that's why I was suggesting a named constant for the error case specifically.
In general I think all deliberate uses of null are somewhat dangerous and prefer that non-nullable be default and enforced with static analysis if possible, so that's mostly what's prompting my suggestion here. "Null References: The Billion Dollar Mistake" is a really good talk going into deeper issues with it. Items 54-55 in Effective Java also touch on problems with null.
My main dislike of deliberate null returns comes down to a combination of two things.
- Nullable values are invisible. It's up to the programmer calling the method to proactively figure out that the return can be null and handle it appropriately, and usually the only tools at their disposal are untrustworthy manually added markers like annotations, comments, etc which can all be out of date, missing, or hard to notice in the first place. The only way to really know is to read the implementation details of every method that returns a potentially nullable type.
- Missing
nullmeans getting an NPE and a program crash, so checking for it is critically important at the same time as being really hard to know when to guard against. NPEs can crop up far removed from the original nullable getter and be generally hard to diagnose.
It's basically breaking down the type system. If any type can be either itself or null, then all code using nullable values needs to first basically do type checking to make sure the value is actually the thing that they're expecting before relying on it like it's an instance of its type. With this Integer case specifically I'd be especially worried about somebody not realizing that this method is nullable and just trying to cast it to int or even comparing it with another int with ==, which would cause an NPE.
These concerns outweigh the semantic harmony of literally returning "nothing" to match that we found "nothing" for me. I'd rather we return a status code labeled "NOT_FOUND" or similar for clients to handle.
Optional<int> is probably the better way to go in a lot of these kinds of cases. It points out that it's possible to get "nothing" back from a method in a deliberate way that's impossible to miss while coding. But I don't think it's available in Android APIs before 24.
shell/platform/android/io/flutter/view/AccessibilityViewEmbedder.java
Outdated
Show resolved
Hide resolved
| * | ||
| * @param embeddedView the embedded view that the record is associated with. | ||
| */ | ||
| public Integer getRecordFlutterId(View embeddedView, AccessibilityRecord record) { |
There was a problem hiding this comment.
I'd either tag this method with @Nullable and document that it returns null when one can't be found or convert it to use int with an INVALID constant for not found cases. It's subjective but I prefer the INVAILD constant option instead since it's more clear and can't potentially end up with NPEs.
There was a problem hiding this comment.
I feel like returning a valid int value here is dangerous as misuse might result in an error that's much harder to spot (setting focus on a wrong a11y node, or setting a wrong child node) vs. an NPE.
There was a problem hiding this comment.
I see the reasoning. I think adding the @Nullable annotations helps with the discoverability here at any rate, so at least now callers of the method have it declared for them.
|
@mklim I tried to reply to your comment about the setters but apparently GitHub doesn't think I'm worthy to respond to that particular comment! You mentioned that there are a deep progression of setters. I would say, yes, on the one hand, those nested setters are boring and tedious. On the other hand, at least they aren't magical/hidden. There are negative aspects of nested control structures, but one positive aspect that they bring is an API that fully recognizes dependencies. The reason that these setters are boring and tedious is because apparently we have a whole bunch of things that need this reference. Maybe the real issue is that we're spreading references around too much? Why is it that so many different objects need this reference? Is it possible that we could do a better job of segmenting role responsibilities, and then end up with far fewer references? I don't want to turn this PR into a prolonged debate on the subject and hold up the work, but I think it would be beneficial for us to ask ourselves questions like these from time to time. Sometimes all we're going to see in a PR is a hint of a larger problem, but if we don't step back to acknowledge the larger problem, then we can't solve it. So, for what it's worth, if it were me implementing this, and I noticed what seemed like an unmanageable number of nested references to the point that I had to introduce a new object just to manage the references, my first inclination would be to look for a deeper reason as to why the references got out of control. When I've done this in the past, I typically end up refactoring a few key relationships that simplifies the whole structure. But that's just my past experience and I don't know for sure if that's how it would work out here or not. So as I mentioned in my earlier comment about this, I'll leave it to Amir to use his judgement in this situations. But hopefully my analysis seems reasonable to both of you. |
| * | ||
| * This is used to propagate an updatable reference to the accessibility bridge within the package. | ||
| */ | ||
| class CurrentAccessibilityBridge { |
There was a problem hiding this comment.
Went with @mklim's suggestion, made this be an AccessibilityEventsDelegate.
We could potentially take this a step further in making sure no-one but PlatformViewsController sets the a11y bridge by splitting this into an interface and an implementation. But I think that this extra complexity isn't justified, as this is an internal class.
| class CurrentAccessibilityBridge { | ||
| private AccessibilityBridge accessibilityBridge; | ||
|
|
||
| public AccessibilityBridge getCurrentAccessibilityBridge() { |
| return accessibilityBridge; | ||
| } | ||
|
|
||
| public void setCurrentAccessibilityBridge(AccessibilityBridge accessibilityBridge) { |
There was a problem hiding this comment.
So this suggestion is to do something like:
class AccessibilityEventsDelegate {
private AccessibilityBridge accessibilityBridge;
AccessibilityEventsDelegate(OnAccessibilityBridgeUpdated onAccessibilityBridgeUpdated) {
onAccessibilityBridgeUpdated.setListener((accessibilityBridge) -> this.accessibilityBridge = accessibilityBridge );
}
}And then have PlatformViewsController to pass in a callback that's updated on attach/detach.
As noted above I don't believe the use circumstances here justifies this kind of code complexity.
|
|
||
| // The accessibility bridge to which accessibility events form the platform views will be dispatched. | ||
| private AccessibilityBridge accessibilityBridge; | ||
| private final CurrentAccessibilityBridge mCurrentAccessibilityBridge; |
There was a problem hiding this comment.
Our codebase right now is in this weird state were part of the newer files are following the Google Style guide and files from up to around 8 months ago are following something similar to the Android style guide.
We should migrate to a single style, but I think it should be done in dedicated PRs, and I wouldn't want to introduce mixed style to this file in this PR. I think we should follow up with a separate PR to rename all the fields here.
| } | ||
| int recordOriginVirtualID = ReflectionAccessors.getVirtualNodeId(recordOriginPackedId); | ||
| Integer recordFlutterId = originToFlutterId.get(new ViewAndId(embeddedView, recordOriginVirtualID)); | ||
| if (recordFlutterId == null) { |
There was a problem hiding this comment.
Changed to check containsKey's return value.
| * | ||
| * @param embeddedView the embedded view that the record is associated with. | ||
| */ | ||
| public Integer getRecordFlutterId(View embeddedView, AccessibilityRecord record) { |
There was a problem hiding this comment.
I feel like returning a valid int value here is dangerous as misuse might result in an error that's much harder to spot (setting focus on a wrong a11y node, or setting a wrong child node) vs. an NPE.
shell/platform/android/io/flutter/view/AccessibilityViewEmbedder.java
Outdated
Show resolved
Hide resolved
|
@matthew-carroll that makes sense and I agree, I think this is uncovering a deeper issue with how we're managing all these class relationships. I also don't want to block the PR on this. It's probably worth filing up a follow up issue to look into this though? At a huge level, we could look at refactoring our code in general to follow some kind of DI pattern. At a smaller level, we could probably clean up the relationships between these classes specifically and try and simplify them if possible. |
|
@mklim SGTM! |
|
@mklim @matthew-carroll can you explain the deep design problem you are talking about? The relationship chain here is: PlatformViewsController --[1 to many]--> VirtualDisplayController -> SingleViewPresentation -> AccessibilityDelegatingFrameLayout PlatformViewsController is what gets the reference to the a11y bridge, and AccessibilityDelegatingFrameLayout is what needs it to send an a11y event. I'd rather not have each AccessibilityDelegatingFrameLayout cache it's own reference to the AccessibilityBridge, this will add boilerplate for setters down the dependency tree, and also generally feels error prone vs. having a single source of truth that's updated with the current a11y bridge. I can see how we can add more protection mechanisms here against one of the clients changing the reference to the a1y bridge, but I think that this will be over engineering in this case as it's internal to a small package. |
|
@amirh there appears to be a particular referential structure, as you laid out in your previous comment. However, it also appears that a reference to the So I don't want to block you on this. As @mklim mentioned, we can come back to this area when none of us are on deadlines and then take a bit more of a conversational and creative investigation into the structure. If we come away from that discussion believing that what we have here is the best we can do, then great, but also maybe we'll discover some beneficial improvements. |
|
@matthew-carroll happy to hear of any concrete improvement suggestion you have when you have them. At a higher level I think the root cause of this complication is that the a11y bridge lifetime is as long as FlutterView is attached to a window, where platform views can outlive being detached from a window(e.g in the theoretical case where FlutterView is moved from one window to another you don't want to lose state for your WebView). I'm not familiar with the a11y bridge enough to know why it's lifetime is such, if that can be changed we can get rid of the indirection here. |
goderbauer
left a comment
There was a problem hiding this comment.
The a11y related stuff looks good. My only concern is around not handling INPUT_FOCUS.
| AccessibilityEvent.TYPE_VIEW_ACCESSIBILITY_FOCUS_CLEARED | ||
| ); | ||
| accessibilityFocusedSemanticsNode = null; | ||
| embeddedFocusedNodeId = null; |
There was a problem hiding this comment.
Is this line actually required? Wouldn't the event to null this out always be handled by the if you added above?
There was a problem hiding this comment.
Discussed offline - we're leaving it here as a defensive measure in case the ACCESSIBLITY_FOCUS_CLEARED event is not sent from the currently accessibility focused node
| if (accessibilityFocusedSemanticsNode != null) { | ||
| return createAccessibilityNodeInfo(accessibilityFocusedSemanticsNode.id); | ||
| } | ||
| if (embeddedFocusedNodeId != null) { |
There was a problem hiding this comment.
What about INPUT_FOCUS for embedded views? Finding the node that has input focus is important for text input via a11y.
There was a problem hiding this comment.
added tracking for input focus as well.
| */ | ||
| @Override | ||
| public boolean performAction(int virtualViewId, int accessibilityAction, @Nullable Bundle arguments) { | ||
| if (virtualViewId >= MIN_ENGINE_GENERATED_NODE_ID) { |
There was a problem hiding this comment.
Do you enforce in the framework that we wrap the id generation at MIN_ENGINE_GENERATED_NODE_ID? If not, please do.
There was a problem hiding this comment.
|
@goderbauer PTAL |
flutter/engine@dd6be2f...af64f72 git log dd6be2f..af64f72 --no-merges --oneline af64f72 Fixing links between higher and lower classes. (flutter/engine#8295) 22ee8ee Migrate existing embedder unit tests to use the fixture. (flutter/engine#8296) 345ae7d Delegate a11y events and action to/from embedded Android platform views. (flutter/engine#8250) The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff ([email protected]), and stop the roller if necessary.
This PR limits the framework generated semantic node IDs to be smaller than 2^16, this allows to safely generate semantic node in the engine with IDs >= 2^16 avoiding ID collision (which is done in flutter/engine#8250).
…ws. (flutter#8250) Delegate a11y events and action to/from embedded Android platfrom views. This handles delegation of: * AccessibilityNodeProvider#performAction * ViewGroup#requestSendAccessibilityEvent * View#onHoverEvent Additionally updates the currently input accessibility focused node state that is tracked by the a11y bridge when an embedded view's node is focused.
…form views. (flutter#8250)" This reverts commit 67bf454.
…form views. (flutter#8250)" This reverts commit 67bf454.
This handles delegation of:
Additionaly updates the curretly accessibility focused node state that is
tracked by the a11y bridge when an embedded view's node is focused.
flutter/flutter#19418