Blizzard / Blizzard/node-rdkafka

How to use seek()/assign() with Kubernetes replicas

Open
#972 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

I've recently had to look into replaying messages from Kafka topics to recover data, the simple option is to have a consumer start from a particular offset/partition using the `seek()` method. However when running at scale on a platform such as Kubernetes using `seek()` becomes a challenge as you are required to know the partition ahead of time when starting up.

I've looked at using `assignments()` to grab the current partition a consumer replica is listening on and then use this data with `seek()`, however I do worry that if a replica drops and the re-balancing takes place that this might result in an inconsistent state.

Is there a reliable way to tell a consumer group to start consuming from a specific offset without having to worry about the details of the partition (in some cases specific consumers will consume from multiple partitions when the replica count is lower than the partition count).

Contributor guide

Open the contributing guide

Research direction

Read the consumer API documentation for seek(), assign(), and assignments(), then compare it with librdkafka's consumer-group offset and rebalance behavior under Kubernetes replicas. Done means documenting whether a group-wide offset reset is supported and the safe recovery procedure, including what happens when partitions are reassigned; no file or test is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, kafka, kubernetes, nodejs
Domain
distributed-systems, infrastructure, stream-processing
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.