Skip to content

Refactor mouse hit testing system: Direct mouse hit test#59883

Merged
fluttergithubbot merged 29 commits intoflutter:masterfrom
dkwingsmt:direct-mouse-hit-test
Jul 7, 2020
Merged

Refactor mouse hit testing system: Direct mouse hit test#59883
fluttergithubbot merged 29 commits intoflutter:masterfrom
dkwingsmt:direct-mouse-hit-test

Conversation

@dkwingsmt
Copy link
Contributor

@dkwingsmt dkwingsmt commented Jun 19, 2020

Description

This PR refactors mouse hit testing so that it uses the regular mouse hit testing system instead of the current layer-based approach. Now hitTestMouseTrackers performs a regular hit testing on the target position, and picks the render objects that inherits MouseTrackerAnnotation.

This change will bring several benefits, including addressing several issues:

What prevented this design from happening in the beginning was performance. Mouse hit testing is done far more frequently than render-object-based regular hit testing, but the render tree is considered too large to be efficiently hit tested.

To resolve this, this PR introduces a new method HitTestResult.pushOffset to allow a few optimizations (introduced in this design doc). Local benchmarks show that the resulting performance is on par with, or even possibly better the layer-based approach.

Macrobenchmarks (Web) Before After
bench_mouse_region_grid_scroll, frame duration 79ms 5ms
bench_mouse_region_grid_hover, frame duration 2.8ms 1.7ms
bench_mouse_region_grid_hover, hit test duration 0.5ms 0.7ms

Related Issues

Besides the issues mentioned above, it also:

Tests

I added the following tests:

  • HitTestResult should correctly push and pop transforms
  • HitTestResult should correctly push and pop offsets
  • AbsorbPointer absorbs pointers
  • IgnorePointer ignores pointers

Changed tests:

  • MouseRegion/Listener's shouldComposite is always false
    • Previously it was true when and only when the widget has any callbacks, cursor, or opaque as true

Checklist

Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I signed the CLA.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I updated/added relevant documentation (doc comments with ///).
  • All existing and new tests are passing.
  • The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
  • I am willing to follow-up on review comments in a timely manner.

Breaking Change

Did any tests fail when you ran them? Please read Handling breaking changes.

@fluttergithubbot fluttergithubbot added framework flutter/packages/flutter repository. See also f: labels. work in progress; do not review labels Jun 19, 2020
@dkwingsmt dkwingsmt marked this pull request as ready for review June 26, 2020 23:29
@fluttergithubbot fluttergithubbot added the f: material design flutter/packages/flutter/material repository. label Jun 27, 2020
@dkwingsmt dkwingsmt added the a: mouse Issues related to using a mouse or mouse support label Jun 27, 2020
@fluttergithubbot fluttergithubbot added the f: material design flutter/packages/flutter/material repository. label Jun 27, 2020
@goderbauer
Copy link
Member

nit: in the PR description:

This PR refactors mouse hit testing so that it uses the regular mouse hit testing system instead of the current layer-based approach.

You probably mean "regular hit testing system on the render tree"?

@goderbauer
Copy link
Member

Also: looks like cirrus had a bad day.

synthesized: event?.synthesized,
transform: event?.transform,
original: event?.original as PointerEnterEvent,
original: null,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we forcing this to null now instead of keeping the original event if available?

Copy link
Contributor Author

@dkwingsmt dkwingsmt Jul 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Short version: In order to prevent the type system complaining, either we set this to null, which satisfies the current definition of original, or we change the type of original in the constructor, which also changes the definition of original.

Long version: An error will be thrown in this as-cast when converting other event types using fromMouseEvent (e.g. from PointerDownEvent to PointerEnterEvent). This as-cast was not here when this constructor was first created, but is reasonable since the constructor requires original to be PointerEnterEvent, which also makes sense since original was added for the transformed constructor to transform between coordinate space. So if we think it this way, this original should be null because original was not defined to suit in the type converting case.

cursor: effectiveMouseCursor,
onEnter: (PointerEnterEvent event) => _handleHover(true),
onExit: (PointerExitEvent event) => _handleHover(false),
return MouseRegion(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we switching this around? Even when it is disabled it should still respond to hovers?

Copy link
Contributor Author

@dkwingsmt dkwingsmt Jul 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. A widget should never turn off the mouse region when it's disabled.

The mouse cursor algorithm detects the first MouseTrackerAnnotation found and uses its cursor. If the MouseRegion is hidden behind the IgnorePointer then the widget will never be able to show its disabled cursor, and the cursor will incorrectly be the one of the widget behind it, as if the TextField does not exist. A test that is violated can be found at

expect(RendererBinding.instance.mouseTracker.debugDeviceActiveCursor(1), SystemMouseCursors.basic);

The reason why this isn't found earlier is probably because IgnorePointer and AbsorbPointer failed to support mouse events (incorrectly, see #35213).

@@ -201,9 +202,17 @@ class RenderView extends RenderObject with RenderObjectWithChildMixin<RenderBox>
// Layer hit testing is done using device pixels, so we have to convert
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we still need to have a separate hit testing pass for mouse events? Can that not be integrated with the regular hit testing now as well?

Copy link
Contributor Author

@dkwingsmt dkwingsmt Jul 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without mouse,

  • During hover events, no regular hit test is done.
  • During down events, a regular hit test is done, so this is an extra pass, but down events happen at such a low frequency that we might not really care.
  • During move events, no regular hit test is done (the result of down events is reused).

Conclusion: The only "extra pass" is the ones at down events, which do not really matter.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am more concerned about the extra code complexity that we have to deal with because the paths are not unified.

Copy link
Contributor Author

@dkwingsmt dkwingsmt Jul 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the mouse device update is triggered by a pointer event, then the code path is indeed more complicated than necessary:


GestureBinding.dispatchEvent -> PointerRouter.route -> MouseTracker._handleEvent -> hit test

Maybe we could've done the hit test at GestureBinding.dispatchEvent and pass it all the way to MouseTracker._handleEvent.

But the mouse device update can also be triggered after every frame, during the post-frame callbacks:

SchedulerBinding.handleDrawFrame -> MouseTracker._updateAllDevices -> hit test

In this case MouseTracker have to initiate a hit test for each device, without the participation of GestureBinding at all.

I'm not saying that a much more unified structure can not be achieved at all, but for now I'd rather touch as few things as possible in order to see the performance impact of the current PR.

Copy link
Member

@goderbauer goderbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great.

_localTransforms.removeLast();
else
_transforms.removeLast();
assert(_transforms.isNotEmpty);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assert is a little odd. Why do we assert after removing that there's still something in _transforms? Shouldn't we assert before?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _transforms should never be empty, because there should at least be the pre-seeded identity matrix.

Of course it shouldn't be empty before the operation either, but that would be detected by removeLast anyway.

@@ -201,9 +202,17 @@ class RenderView extends RenderObject with RenderObjectWithChildMixin<RenderBox>
// Layer hit testing is done using device pixels, so we have to convert
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am more concerned about the extra code complexity that we have to deal with because the paths are not unified.

Copy link
Member

@goderbauer goderbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ferhatb
Copy link
Contributor

ferhatb commented Jul 7, 2020

This is breaking customer app, please see #60995

@liyuqian liyuqian added the c: performance Relates to speed or footprint issues (see "perf:" labels) label Jul 9, 2020
mingwandroid pushed a commit to mingwandroid/flutter that referenced this pull request Sep 6, 2020
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 30, 2021
@flutter-dashboard flutter-dashboard bot added a: text input Entering text in a text field or keyboard related problems f: gestures flutter/packages/flutter/gestures repository. f: scrolling Viewports, list views, slivers, etc. labels Sep 28, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

a: mouse Issues related to using a mouse or mouse support a: text input Entering text in a text field or keyboard related problems c: performance Relates to speed or footprint issues (see "perf:" labels) f: gestures flutter/packages/flutter/gestures repository. f: material design flutter/packages/flutter/material repository. f: scrolling Viewports, list views, slivers, etc. framework flutter/packages/flutter repository. See also f: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move annotations to a separate tree Poor performance when subscribing to MouseRegion events IgnorePointer does not prevent Listener.hover events

7 participants