Skip to content

add gradient benchmarks#101549

Merged
flar merged 1 commit intoflutter:masterfrom
flar:add-gradient-benchmarks
Apr 9, 2022
Merged

add gradient benchmarks#101549
flar merged 1 commit intoflutter:masterfrom
flar:add-gradient-benchmarks

Conversation

@flar
Copy link
Contributor

@flar flar commented Apr 8, 2022

We are seeing a slight performance regression on some downstream projects on an engine commit that changed the way we store paint shaders (image, gradient, etc.) and it looks like we don't have any benchmarks to track the performance of our gradient shaders. This PR adds 3 gradient benchmarks:

  • "dynamic" which creates a bunch of new gradients every frame that are constantly changing
  • "consistent" which creates new gradient objects on every frame, but with the same contents each frame
  • "static" which statically allocates its gradient objects for reuse on every frame

@flar flar requested a review from keyonghan as a code owner April 8, 2022 01:10
@flutter-dashboard flutter-dashboard bot added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Apr 8, 2022
@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@flar
Copy link
Contributor Author

flar commented Apr 8, 2022

@dnfield Can you run these benchmarks on the target devices?

@flar
Copy link
Contributor Author

flar commented Apr 8, 2022

Hmm, shouldn't benchmarks be exempt from the test check robots?

Copy link
Contributor

@dnfield dnfield left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

The bot doesn't understand these file patterns - feel free to update it at flutter/cocoon.

@Hixie for test exemption

@dnfield
Copy link
Contributor

dnfield commented Apr 8, 2022

I'll run these on the low end device next time I'm in the office

@fluttergithubbot
Copy link
Contributor

This pull request is not suitable for automatic merging in its current state.

  • The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label.

Copy link
Contributor

@keyonghan keyonghan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@flar flar added waiting for tree to go green and removed c: contributor-productivity Team-specific productivity, code health, technical debt. labels Apr 8, 2022
@fluttergithubbot
Copy link
Contributor

This pull request is not suitable for automatic merging in its current state.

  • The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label.

@zanderso
Copy link
Member

zanderso commented Apr 8, 2022

Related: #101518

@flar
Copy link
Contributor Author

flar commented Apr 9, 2022

Here are the results of doing an A/B run on these benchmarks against a pre-DlColorSource engine. For each of these results, the default, or A engine is using the DlColorSource and the local, or B engine is using SkGradient. The "Speed-up" column is how much faster Sk is against Dl (or slower if the factor is less than 1.0).

Results for the "static" version of the benchmark:

═══════════╡ ••• Formatted summary for "ABresults.static.json" ••• ╞════════════

                   Score                     Average A (noise)  Average B (noise)   Speed-up
average_frame_build_time_millis                   5.35 (0.59%)       4.45 (0.92%)    1.20x  
worst_frame_build_time_millis                     7.79 (0.64%)       7.25 (11.94%)   1.07x  
90th_percentile_frame_build_time_millis           5.80 (0.86%)       4.90 (0.27%)    1.18x  
99th_percentile_frame_build_time_millis           6.36 (6.13%)       5.52 (3.56%)    1.15x  
average_frame_rasterizer_time_millis             15.15 (0.41%)      16.01 (0.91%)    0.95x  
worst_frame_rasterizer_time_millis               18.85 (2.28%)      20.16 (7.17%)    0.94x  
90th_percentile_frame_rasterizer_time_millis     15.43 (0.61%)      16.28 (1.25%)    0.95x  
99th_percentile_frame_rasterizer_time_millis     16.04 (1.37%)      17.55 (2.76%)    0.91x  
average_layer_cache_count                         0.00 (0.00%)       0.00 (0.00%)     NaNx  
90th_percentile_layer_cache_count                 0.00 (0.00%)       0.00 (0.00%)     NaNx  
99th_percentile_layer_cache_count                 0.00 (0.00%)       0.00 (0.00%)     NaNx  
worst_layer_cache_count                           0.00 (0.00%)       0.00 (0.00%)     NaNx  
average_layer_cache_memory                        0.00 (0.00%)       0.00 (0.00%)     NaNx  
90th_percentile_layer_cache_memory                0.00 (0.00%)       0.00 (0.00%)     NaNx  
99th_percentile_layer_cache_memory                0.00 (0.00%)       0.00 (0.00%)     NaNx  
worst_layer_cache_memory                          0.00 (0.00%)       0.00 (0.00%)     NaNx  
average_picture_cache_count                       0.00 (0.00%)       0.00 (0.00%)     NaNx  
90th_percentile_picture_cache_count               0.00 (0.00%)       0.00 (0.00%)     NaNx  
99th_percentile_picture_cache_count               0.00 (0.00%)       0.00 (0.00%)     NaNx  
worst_picture_cache_count                         0.00 (0.00%)       0.00 (0.00%)     NaNx  
average_picture_cache_memory                      0.00 (0.00%)       0.00 (0.00%)     NaNx  
90th_percentile_picture_cache_memory              0.00 (0.00%)       0.00 (0.00%)     NaNx  
99th_percentile_picture_cache_memory              0.00 (0.00%)       0.00 (0.00%)     NaNx  
worst_picture_cache_memory                        0.00 (0.00%)       0.00 (0.00%)     NaNx  
new_gen_gc_count                                  5.50 (15.75%)      5.00 (20.00%)   1.10x  
old_gen_gc_count                                  0.00 (0.00%)       1.00 (100.00%)  0.00x  

And the results for the "consistent" version of the benchmark:

═════════╡ ••• Formatted summary for "ABresults.consistent.json" ••• ╞══════════

                   Score                     Average A (noise)  Average B (noise)   Speed-up
average_frame_build_time_millis                  10.42 (1.20%)      10.46 (1.11%)    1.00x  
worst_frame_build_time_millis                    74.88 (1.36%)      62.10 (3.12%)    1.21x  
90th_percentile_frame_build_time_millis          10.86 (0.57%)      11.08 (0.42%)    0.98x  
99th_percentile_frame_build_time_millis          24.14 (89.89%)     11.99 (1.11%)    2.01x  
average_frame_rasterizer_time_millis             16.03 (0.45%)      16.60 (0.33%)    0.97x  
worst_frame_rasterizer_time_millis               27.83 (11.09%)     24.06 (3.36%)    1.16x  
90th_percentile_frame_rasterizer_time_millis     16.57 (0.42%)      17.03 (1.30%)    0.97x  
99th_percentile_frame_rasterizer_time_millis     20.68 (5.65%)      21.17 (1.55%)    0.98x  
average_layer_cache_count                         0.00 (0.00%)       0.00 (0.00%)     NaNx  
90th_percentile_layer_cache_count                 0.00 (0.00%)       0.00 (0.00%)     NaNx  
99th_percentile_layer_cache_count                 0.00 (0.00%)       0.00 (0.00%)     NaNx  
worst_layer_cache_count                           0.00 (0.00%)       0.00 (0.00%)     NaNx  
average_layer_cache_memory                        0.00 (0.00%)       0.00 (0.00%)     NaNx  
90th_percentile_layer_cache_memory                0.00 (0.00%)       0.00 (0.00%)     NaNx  
99th_percentile_layer_cache_memory                0.00 (0.00%)       0.00 (0.00%)     NaNx  
worst_layer_cache_memory                          0.00 (0.00%)       0.00 (0.00%)     NaNx  
average_picture_cache_count                       0.00 (0.00%)       0.00 (0.00%)     NaNx  
90th_percentile_picture_cache_count               0.00 (0.00%)       0.00 (0.00%)     NaNx  
99th_percentile_picture_cache_count               0.00 (0.00%)       0.00 (0.00%)     NaNx  
worst_picture_cache_count                         0.00 (0.00%)       0.00 (0.00%)     NaNx  
average_picture_cache_memory                      0.00 (0.00%)       0.00 (0.00%)     NaNx  
90th_percentile_picture_cache_memory              0.00 (0.00%)       0.00 (0.00%)     NaNx  
99th_percentile_picture_cache_memory              0.00 (0.00%)       0.00 (0.00%)     NaNx  
worst_picture_cache_memory                        0.00 (0.00%)       0.00 (0.00%)     NaNx  
new_gen_gc_count                                 28.00 (0.00%)      16.00 (12.50%)   1.75x  
old_gen_gc_count                                  0.00 (0.00%)       0.50 (173.21%)  0.00x  

And results for the "dynamic" version of the benchmark:

═══════════╡ ••• Formatted summary for "ABresults.dynamic.json" ••• ╞═══════════

                   Score                     Average A (noise)  Average B (noise)  Speed-up
average_frame_build_time_millis                  10.36 (0.91%)      10.36 (0.95%)   1.00x  
worst_frame_build_time_millis                    75.79 (2.26%)      59.28 (3.59%)   1.28x  
90th_percentile_frame_build_time_millis          10.88 (0.62%)      11.10 (0.13%)   0.98x  
99th_percentile_frame_build_time_millis          12.15 (3.12%)      20.73 (72.73%)  0.59x  
average_frame_rasterizer_time_millis             16.07 (0.43%)      16.55 (0.34%)   0.97x  
worst_frame_rasterizer_time_millis               25.29 (13.25%)     25.49 (4.35%)   0.99x  
90th_percentile_frame_rasterizer_time_millis     16.65 (0.70%)      16.92 (0.39%)   0.98x  
99th_percentile_frame_rasterizer_time_millis     20.61 (4.29%)      21.40 (2.24%)   0.96x  
average_layer_cache_count                         0.00 (0.00%)       0.00 (0.00%)    NaNx  
90th_percentile_layer_cache_count                 0.00 (0.00%)       0.00 (0.00%)    NaNx  
99th_percentile_layer_cache_count                 0.00 (0.00%)       0.00 (0.00%)    NaNx  
worst_layer_cache_count                           0.00 (0.00%)       0.00 (0.00%)    NaNx  
average_layer_cache_memory                        0.00 (0.00%)       0.00 (0.00%)    NaNx  
90th_percentile_layer_cache_memory                0.00 (0.00%)       0.00 (0.00%)    NaNx  
99th_percentile_layer_cache_memory                0.00 (0.00%)       0.00 (0.00%)    NaNx  
worst_layer_cache_memory                          0.00 (0.00%)       0.00 (0.00%)    NaNx  
average_picture_cache_count                       0.00 (0.00%)       0.00 (0.00%)    NaNx  
90th_percentile_picture_cache_count               0.00 (0.00%)       0.00 (0.00%)    NaNx  
99th_percentile_picture_cache_count               0.00 (0.00%)       0.00 (0.00%)    NaNx  
worst_picture_cache_count                         0.00 (0.00%)       0.00 (0.00%)    NaNx  
average_picture_cache_memory                      0.00 (0.00%)       0.00 (0.00%)    NaNx  
90th_percentile_picture_cache_memory              0.00 (0.00%)       0.00 (0.00%)    NaNx  
99th_percentile_picture_cache_memory              0.00 (0.00%)       0.00 (0.00%)    NaNx  
worst_picture_cache_memory                        0.00 (0.00%)       0.00 (0.00%)    NaNx  
new_gen_gc_count                                 30.50 (14.94%)     15.50 (10.70%)  1.97x  
old_gen_gc_count                                  0.00 (0.00%)       0.00 (0.00%)    NaNx  

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants