Blizzard / Blizzard/node-rdkafka

Consumer is picking messages from single partition after adding 3 hours of delayed in consuming.

Open
#1,045 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

**Environment Information**
- OS [e.g. Mac, Arch, Windows 10]: Mac OS 13
- Node Version [e.g. 8.2.1]: 12.14.1
- NPM Version [e.g. 5.4.2]:6.13.4
- C++ Toolchain [e.g. Visual Studio, llvm, g++]:
- node-rdkafka version [e.g. 2.3.3]: 2.7.4

```
this.consumer = new Kafka.KafkaConsumer(
{
'group.id': consumerConfig.defaultGroupId + "_" + topics.join("_"),
'client.id': consumerConfig.defaultClientId + "_" + topics.join("_"),
'metadata.broker.list': consumerConfig.brokerList,
'compression.codec': consumerConfig.compressionCodec,
'retry.backoff.ms': 100,
'socket.keepalive.enable': true,
'enable.auto.commit': this.autoCommit,
'statistics.interval.ms': 5000,
// 'queued.min.messages': 10,
// 'fetch.message.max.bytes': 30,
// 'queued.max.messages.kbytes': 10 // didn't work
debug: 'all'
},
{
'auto.offset.reset': consumerConfig.autoOffsetReset,
'request.required.acks': consumerConfig.requestRequiredAcks
}
);

```

Topic: profile_updates
Partitions: 10

We are trying to add a mechanism where we produce msgs at 5-6 qps but we want to consume those msgs after 3 hours of delay since produce.
We consume in a batch of 90 messages. Each msg of ~150 bytes.
In the first batch, we checked the timestamp and add delay of
`3 hrs - (Date.now()- msg_produce_timestamp)` i.e slept for ~3 hours.
We are using manual commit (autoCommit: false).

Now the issue with this workflow is, after waking up from sleep, the consumer is picking all 90 msgs from partition:0 only.
As a result, lag keeps increasing in the other 9 partitions.
Is there any way/configuration to optimize this flow so that we could consume from each partition equally and hence the lag will be the same in all partitions?

Contributor guide

Open the contributing guide

Research direction

Start with the Kafka.KafkaConsumer setup and manual-commit batch workflow shown in the issue; compare how the consumer handles a three-hour sleep and partition assignment. Reproduce the behavior with the 10-partition profile_updates topic and 90-message batches, then verify that consumption and lag are balanced across partitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, kafka, nodejs
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.