Allow replicas to send some buckets out of order#80179
Merged
Conversation
Contributor
clickhouse-gh
bot
added
the
pr-not-for-changelog
clickhouse-gh
bot
added
pr-performance
nickitat
force-pushed
the
out_of_order_buckets
branch
from
22d84e9 to
407e14d
Compare
Member
Author
devcrafter
approved these changes
Aug 5, 2025
| ) | ||
|
|
||
| # big number of granules + low total size in bytes = super tiny granules = big min_marks_per_task | ||
| # => big mark_segment_size will be chosen. it is not required to be big, just not equal to the default |
Member
There was a problem hiding this comment.
What this comment refers to in the test? I don't see we set any settings for MergeTree
nickitat
commented
Aug 5, 2025
tests/integration/test_backward_compatibility/test_aggregation_with_out_of_order_buckets.py
Outdated
Show resolved
Hide resolved
Member
Author
robot-ch-test-poll4
added
the
pr-synced-to-cloud
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 14, 2025
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 15, 2025
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 16, 2025
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 18, 2025
Member
Author
github-merge-queue bot
pushed a commit
that referenced
this pull request
Aug 19, 2025
Revert "Merge pull request #80179 from ClickHouse/out_of_order_buckets"
Merged
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added new logic (controlled by the setting
enable_producing_buckets_out_of_order_in_aggregation, enabled by default) that allows sending some buckets out of order during memory-efficient aggregation. When some aggregation buckets take significantly longer to merge than others, it improves performance by allowing the initiator to merge buckets with higher bucket id-s in the meantime. The downside is potentially higher memory usage (shouldn't be significant).Motivation was described here: https://github.com/ClickHouse/clickhouse-private/issues/22937#issuecomment-2731022465
All the essence is in
src/Processors/Transforms/AggregatingTransform.cppandsrc/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp.