Blizzard / Blizzard/node-rdkafka
Consuming messages from each partition separately
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
A Consumer that is subscribed to multiple partitions cannot control the mix of messages it consumes from each partition.
There are various reasons why controlling that is relevant, see eg. #719 and https://github.com/confluentinc/librdkafka/wiki/FAQ#what-are-partition-queues-and-why-are-some-partitions-slower-than-others.
That last link shows that librdkafka supports consuming from each partition separately. It would be great if that implementation is available in node-rdkafka as well. An example could be a `PartitionConsumer` class, that does not have the normal `.consume` methods, but `.consume(topic, partition, number, callback)`.
Contributor guide
Research direction
Start with the existing Consumer entry points and the librdkafka partition-queues FAQ, then review the context in #719. Define how a PartitionConsumer would expose `.consume(topic, partition, number, callback)` and verify that consumers subscribed to multiple partitions can control each partition independently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kafka, nodejs
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100