Blizzard / Blizzard/node-rdkafka
Consumer is picking messages from single partition after adding 3 hours of delayed in consuming.
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
**Environment Information**
- OS [e.g. Mac, Arch, Windows 10]: Mac OS 13
- Node Version [e.g. 8.2.1]: 12.14.1
- NPM Version [e.g. 5.4.2]:6.13.4
- C++ Toolchain [e.g. Visual Studio, llvm, g++]:
- node-rdkafka version [e.g. 2.3.3]: 2.7.4
```
this.consumer = new Kafka.KafkaConsumer(
{
'group.id': consumerConfig.defaultGroupId + "_" + topics.join("_"),
'client.id': consumerConfig.defaultClientId + "_" + topics.join("_"),
'metadata.broker.list': consumerConfig.brokerList,
'compression.codec': consumerConfig.compressionCodec,
'retry.backoff.ms': 100,
'socket.keepalive.enable': true,
'enable.auto.commit': this.autoCommit,
'statistics.interval.ms': 5000,
// 'queued.min.messages': 10,
// 'fetch.message.max.bytes': 30,
// 'queued.max.messages.kbytes': 10 // didn't work
debug: 'all'
},
{
'auto.offset.reset': consumerConfig.autoOffsetReset,
'request.required.acks': consumerConfig.requestRequiredAcks
}
);
```
Topic: profile_updates
Partitions: 10
We are trying to add a mechanism where we produce msgs at 5-6 qps but we want to consume those msgs after 3 hours of delay since produce.
We consume in a batch of 90 messages. Each msg of ~150 bytes.
In the first batch, we checked the timestamp and add delay of
`3 hrs - (Date.now()- msg_produce_timestamp)` i.e slept for ~3 hours.
We are using manual commit (autoCommit: false).
Now the issue with this workflow is, after waking up from sleep, the consumer is picking all 90 msgs from partition:0 only.
As a result, lag keeps increasing in the other 9 partitions.
Is there any way/configuration to optimize this flow so that we could consume from each partition equally and hence the lag will be the same in all partitions?
Contributor guide
Research direction
Start with the Kafka.KafkaConsumer setup and manual-commit batch workflow shown in the issue; compare how the consumer handles a three-hour sleep and partition assignment. Reproduce the behavior with the 10-partition profile_updates topic and 90-message batches, then verify that consumption and lag are balanced across partitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kafka, nodejs
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100