Blizzard / Blizzard/node-rdkafka

Consuming messages from each partition separately

Open
#1,063 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

A Consumer that is subscribed to multiple partitions cannot control the mix of messages it consumes from each partition.
There are various reasons why controlling that is relevant, see eg. #719 and https://github.com/confluentinc/librdkafka/wiki/FAQ#what-are-partition-queues-and-why-are-some-partitions-slower-than-others.

That last link shows that librdkafka supports consuming from each partition separately. It would be great if that implementation is available in node-rdkafka as well. An example could be a `PartitionConsumer` class, that does not have the normal `.consume` methods, but `.consume(topic, partition, number, callback)`.

Contributor guide

Open the contributing guide

Research direction

Start with the existing Consumer entry points and the librdkafka partition-queues FAQ, then review the context in #719. Define how a PartitionConsumer would expose `.consume(topic, partition, number, callback)` and verify that consumers subscribed to multiple partitions can control each partition independently.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, kafka, nodejs
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.