commitOffsets can be passed the offsets to commit#10
Closed
squito wants to merge 2 commits intoapache:0.8from
Closed
commitOffsets can be passed the offsets to commit#10squito wants to merge 2 commits intoapache:0.8from
squito wants to merge 2 commits intoapache:0.8from
Conversation
Contributor
|
@squito AFAIK code improvements are only accepted in a form of jira ticket + attached patch file. Not absolutely sure what is the official policy, but looks like this mirror is kept only for code navigation/personal forks purpose. |
Author
|
thanks @lazyval . I've created this ticket https://issues.apache.org/jira/browse/KAFKA-1144 . |
Closed
ymatsuda
referenced
this pull request
in confluentinc/kafka
Aug 5, 2015
ymatsuda
pushed a commit
to ymatsuda/kafka
that referenced
this pull request
Aug 27, 2015
kstream refactored
Parth-Brahmbhatt
pushed a commit
to Parth-Brahmbhatt/kafka
that referenced
this pull request
Nov 3, 2015
BUG-45592: Migration script for Upgrading old acls to new acls.
xiowu0
pushed a commit
to xiowu0/kafka
that referenced
this pull request
Apr 2, 2019
…s Java objects as arguments. (apache#10) Reviewers: Radai Rosenblatt, Ke Hu
abhishekmendhekar
pushed a commit
to abhishekmendhekar/kafka
that referenced
this pull request
Jun 12, 2019
…ce only takes Java objects as arguments. (apache#10) TICKET = LI_DESCRIPTION = Reviewers: Radai Rosenblatt, Ke Hu EXIT_CRITERIA = MANUAL ["describe exit criteria"]
efeg
pushed a commit
to efeg/kafka
that referenced
this pull request
Jan 29, 2020
…md (apache#10) Clean up metrics aggregator. Improve quick start procedure in README.md Fixed a bug in the MetricCompletenessChecker
lianetm
added a commit
to lianetm/kafka
that referenced
this pull request
Jun 2, 2023
* beginningOffsets and endOffsets implementation * ListOffsetsRequestManager implementation & event processing integration
lianetm
added a commit
to lianetm/kafka
that referenced
this pull request
Jun 12, 2023
* beginningOffsets and endOffsets implementation * ListOffsetsRequestManager implementation & event processing integration
brandboat
pushed a commit
to brandboat/kafka
that referenced
this pull request
Sep 15, 2025
…tadata-manager Add remote cluster metadata manager
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds another version of
commitOffsetsthat takes the offsets to commit as a parameter.Without this change, getting correct user code is very hard. Despite kafka's at-least-once guarantees, most user code doesn't actually have that guarantee, and is almost certainly wrong if doing batch processing. Getting it right requires some very careful synchronization between all consumer threads, which is both:
This small change simplifies a lot of this. This was discussed extensively on the user mailing list, on the thread "are kafka consumer apps guaranteed to see msgs at least once?"
You can also see an example implementation of a user api which makes use of this, to get proper at-least-once guarantees by user code, even for batches:
https://github.com/quantifind/kafka-utils/pull/1
I'm open to any suggestions on how to add unit tests for this.