Blizzard / Blizzard/node-rdkafka
How to use seek()/assign() with Kubernetes replicas
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
I've recently had to look into replaying messages from Kafka topics to recover data, the simple option is to have a consumer start from a particular offset/partition using the `seek()` method. However when running at scale on a platform such as Kubernetes using `seek()` becomes a challenge as you are required to know the partition ahead of time when starting up.
I've looked at using `assignments()` to grab the current partition a consumer replica is listening on and then use this data with `seek()`, however I do worry that if a replica drops and the re-balancing takes place that this might result in an inconsistent state.
Is there a reliable way to tell a consumer group to start consuming from a specific offset without having to worry about the details of the partition (in some cases specific consumers will consume from multiple partitions when the replica count is lower than the partition count).
Contributor guide
Research direction
Read the consumer API documentation for seek(), assign(), and assignments(), then compare it with librdkafka's consumer-group offset and rebalance behavior under Kubernetes replicas. Done means documenting whether a group-wide offset reset is supported and the safe recovery procedure, including what happens when partitions are reassigned; no file or test is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kafka, kubernetes, nodejs
- Domain
- distributed-systems, infrastructure, stream-processing
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100