Simple kafka caching#431
Open
Yashwanth-Ranjan-Singaravel wants to merge 20 commits intoopen-lambda:mainfrom
Open
Simple kafka caching#431Yashwanth-Ranjan-Singaravel wants to merge 20 commits intoopen-lambda:mainfrom
Yashwanth-Ranjan-Singaravel wants to merge 20 commits intoopen-lambda:mainfrom
Conversation
… to use this field
tylerharter
requested changes
Mar 20, 2026
| lambdaName string // lambda function name | ||
| kafkaTrigger *common.KafkaTrigger | ||
| client KafkaClient // used for PollFetches/Close (may be a cachedKafkaClient) | ||
| cache *cachedKafkaClient // typed reference for Seek; same object as client when caching is enabled |
Member
There was a problem hiding this comment.
No cache entry in this struct. Just client. The client may be a cachedKafkaClient that wraps a regular client.
| "size", len(record.Value)) | ||
| lkc.processMessage(record) | ||
| }) | ||
| if seek := lkc.processMessage(record); seek != nil && lkc.cache != nil { |
Member
There was a problem hiding this comment.
At this level of abstraction, we shouldn't be thinking about caching (even though it is helping us). We should seek the client the same, regular it is a raw client, or a cache client that wraps a raw client.
| // cachedKafkaClient wraps a KafkaClient and caches records in an LRU map keyed | ||
| // by {topic, partition, offset}. When a seek is active, PollFetches serves | ||
| // records from the cache. On cache miss, the seek ends and normal polling resumes. | ||
| type cachedKafkaClient struct { |
Member
There was a problem hiding this comment.
Move cachedKafkaClient to another file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a very simple kafka seek + caching mechanism