Skip to content

Kafka fixes part2#8917

Merged
alexey-milovidov merged 3 commits intoClickHouse:masterfrom
filimonov:kafka_fixes_part2
Jan 31, 2020
Merged

Kafka fixes part2#8917
alexey-milovidov merged 3 commits intoClickHouse:masterfrom
filimonov:kafka_fixes_part2

Conversation

@filimonov
Copy link
Contributor

@filimonov filimonov commented Jan 30, 2020 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • Bug Fix

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

  1. fix duplicates that were appearing during consumer group rebalance. Fixes Kafka: duplicates on consumer group rebalance  #7259
  2. fix rare 'holes' appeared when data were polled from several partitions with one poll and committed partially (now we always process / commit the whole polled block of messages)
  3. fix flushes by block size (before that only flushing by timeout was working properly).
  4. better subscription procedure (with assignment feedback)
  5. make tests work faster (with default intervals and timeouts)

Due to the fact that data was not flushed by block size before (as it should according to documentation), that PR may lead to some performance degradation with default settings (due to more often & tinier flushes which are less optimal). If you encounter the performance issue after that change - please increase kafka_max_block_size in the table to the bigger value ( for example CREATE TABLE ...Engine=Kafka ... SETTINGS ... kafka_max_block_size=524288).

Detailed description / Documentation draft:

Supersedes #8876

@filimonov filimonov mentioned this pull request Jan 30, 2020
Closed
@alexey-milovidov alexey-milovidov merged commit 5d3c62c into ClickHouse:master Jan 31, 2020
@filimonov
Copy link
Contributor Author

filimonov commented Jan 31, 2020 •
edited
Loading

Performance details:

# baseline before that PR
┌─version()──┬─block_size─┬─consume_time─┬─consume_speed─┐
│ 20.2.1.2269 │      65536 │          122 │ 137518.163934 │
│ 20.2.1.2269 │     131072 │          120 │ 139810.133333 │
│ 20.2.1.2269 │     524288 │          116 │ 144631.172413 │
│ 20.2.1.2269 │    1048576 │          112 │ 149796.571428 │

# after that PR
┌─version()──┬─block_size─┬─consume_time─┬─consume_speed─┐
│ 20.2.1.2270 │      65536 │          163 │ 102927.705521 │ <- default value, 25% slower 
│ 20.2.1.2270 │     131072 │          128 │ 131072.000000 │ <- 128K blocks, 6% slower
│ 20.2.1.2270 │     524288 │          107 │ 156796.411214 │ <- 512K blocks, 8% faster (or 14% faster than baseline with default settings)
│ 20.2.1.2270 │    1048576 │          102 │ 164482.509803 │ <- 1M blocks, 10% faster

# both tested with `stream_poll_timeout_ms =100, stream_flush_interval_ms = 800`

@filimonov filimonov added the pr-bugfix Pull request with bugfix, not backported by default label Jan 31, 2020
This was referenced Feb 4, 2020
Merged
Merged
Merged
@filimonov filimonov mentioned this pull request Mar 7, 2020
Closed
@filimonov filimonov mentioned this pull request May 27, 2020
Merged
@filimonov filimonov mentioned this pull request Jun 4, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp-kafka Kafka Engine pr-bugfix Pull request with bugfix, not backported by default

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Kafka: duplicates on consumer group rebalance

3 participants

@filimonov @tavplubix @alexey-milovidov