Skip to content

Optimized hdr_value_at_percentile#10606

Merged
oranagra merged 4 commits intoredis:unstablefrom
yoav-steinberg:hdr_hist_optimization
Apr 20, 2022
Merged

Optimized hdr_value_at_percentile#10606
oranagra merged 4 commits intoredis:unstablefrom
yoav-steinberg:hdr_hist_optimization

Conversation

@yoav-steinberg
Copy link
Copy Markdown
Contributor

@yoav-steinberg yoav-steinberg commented Apr 19, 2022

hdr_value_at_percentile() is part of the Hdr_Histogram library used when generating latencystats report.

There's a pending optimization for this function which greatly affects the performance of info latencystats.

This PR:

  1. Upgrades the sources in deps/hdr_histogram to the latest Hdr_Histogram version 0.11.5
  2. Applies the referenced optimization.
  3. Adds minor documentation about the hdr_histogram dependency which was missing under deps/README.md.

benchmark on my machine:
running: redis-benchmark -n 100000 info latencystats on a clean build with no data.

benchmark RPS
before upgrade to v0.11.05 7,681
before optimization 12,474
after optimization 52,606

@yoav-steinberg yoav-steinberg force-pushed the hdr_hist_optimization branch from af88237 to 51d9c93 Compare April 19, 2022 13:16
Comment thread deps/README.md Outdated
Comment thread deps/hdr_histogram/hdr_tests.h
Comment thread deps/hdr_histogram/hdr_alloc.h
Comment thread deps/hdr_histogram/Makefile
Co-authored-by: Oran Agra <[email protected]>
Copy link
Copy Markdown
Member

@oranagra oranagra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i didn't review the actual changes to the hdr_histogram code itself, but then again, i didn't review these when they where introduced into redis either.
leaving this PR open for a while to see if anyone else wants to comment.

@filipecosta90
Copy link
Copy Markdown
Contributor

i didn't review the actual changes to the hdr_histogram code itself, but then again, i didn't review these when they where introduced into redis either.

@oranagra the changes from @yoav-steinberg are fully safe and have been prooveed to be working as expected also on the go version of the hdrhistogram. IMHO we can merge this improvement without waiting for the merge of HdrHistogram/HdrHistogram_c#107.

excellent work @yoav-steinberg and thank you for pushing this forward! Impressive to see the bump in the INFO LATENCYSTATS command.

@oranagra oranagra merged commit 5075e74 into redis:unstable Apr 20, 2022
@yangbodong22011
Copy link
Copy Markdown
Contributor

@oranagra
Copy link
Copy Markdown
Member

@yangbodong22011 we don't.. see -DHDR_MALLOC_INCLUDE=\"hdr_redis_malloc.h\"
the point here is to avoid modifications on the files we take from upstream.
maybe we can think of a way to add a regression test for this?

@yangbodong22011
Copy link
Copy Markdown
Contributor

the point here is to avoid modifications on the files we take from upstream.

oh, I didn't check the history, it turns out this is already part of the library itself.

IMHO the correct way to define an allocator should be through an interface, not by modifying files: https://github.com/RoaringBitmap/CRoaring/pull/358/files

@oranagra oranagra mentioned this pull request Apr 27, 2022
yoav-steinberg added a commit to yoav-steinberg/redis that referenced this pull request May 20, 2022
…#10606.

The code is based on upstream https://github.com/HdrHistogram/HdrHistogram_c master branch latest commit (e4448cf6d1cd08fff519812d3b1e58bd5a94ac42). The reason to pull this in now is that their final version of our optimization is even faster. See: HdrHistogram/HdrHistogram_c#107.
oranagra pushed a commit that referenced this pull request May 22, 2022
… (#10755)

The code is based on upstream https://github.com/HdrHistogram/HdrHistogram_c
master branch latest commit (e4448cf6d1cd08fff519812d3b1e58bd5a94ac42).
The reason to pull this in now is that their final version of our optimization is even faster.
See: HdrHistogram/HdrHistogram_c#107.
enjoy-binbin pushed a commit to enjoy-binbin/redis that referenced this pull request Jul 31, 2023
`hdr_value_at_percentile()` is part of the Hdr_Histogram library
used when generating `latencystats` report. 

There's a pending optimization for this function which greatly
affects the performance of `info latencystats`.
HdrHistogram/HdrHistogram_c#107

This PR:
1. Upgrades the sources in _deps/hdr_histogram_ to the latest Hdr_Histogram
  version 0.11.5
2. Applies the referenced optimization.
3. Adds minor documentation about the hdr_histogram dependency which was
  missing under _deps/README.md_.

benchmark on my machine:
running: `redis-benchmark -n 100000 info latencystats` on a clean build with no data.

| benchmark | RPS |
| ---- | ---- |
| before upgrade to v0.11.05  | 7,681 |
| before optimization | 12,474 |
| after optimization | 52,606 |

Co-authored-by: filipe oliveira <[email protected]>
enjoy-binbin pushed a commit to enjoy-binbin/redis that referenced this pull request Jul 31, 2023
…#10606. (redis#10755)

The code is based on upstream https://github.com/HdrHistogram/HdrHistogram_c
master branch latest commit (e4448cf6d1cd08fff519812d3b1e58bd5a94ac42).
The reason to pull this in now is that their final version of our optimization is even faster.
See: HdrHistogram/HdrHistogram_c#107.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

4 participants