Blizzard / Blizzard/node-rdkafka
The consumer does not consume all partitions fairly
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
**Environment Information**
- OS [CentOS Linux]:
- Node Version [12 and 14]:
- NPM Version [6.14.5]:
- node-rdkafka version [2.9.0]:
**node-rdkafka Configuration Settings**
I use non-following mode consumer and each batch contains 2000 records, consumer configuration:
```
{
groupId: 'frg',
bootstrapServer: '10.120.1.20:9092',
topic: 'rdo-stocks-v1',
clientId: 'rdo-stock-quantity',
enableAutoCommit: true,
enableAutoOffsetStore: true,
socketKeepaliveEnable: 'true',
securityProtocol: 'SASL_PLAINTEXT',
saslMechanisms: 'SCRAM-SHA-512',
saslUsername: 'rdUser',
saslPassword: 'secret',
consumerMaxPollInterval: '200000',
consumerAutoOffsetReset: 'earliest',
consumerCommitInterval: 500,
consumerPartitionAssignmentStrategy: 'roundrobin',
debug: 'fetch',
consumerHeartbeatInternalMs: 10000,
consumerSessionTimeout: 20000,
consumerFetchMessageSizeByte: 1048576,
messageMaxBytes: 1548576,
consumerQueuedMaxMessagesKbytes: 1048576,
consumerMaxPollRecords: 2000,
consumerDefaultTimeout: 100
}
```
UV_THREADPOOL_SIZE is set to 30.
The consumer run on in a docker container and each consumer runs on a different host (8 cores).
**Additional context**
The consumer consumes messages well, but as we can see in the picture below, each Kafka broker is the leader of two partitions and the consumer consumes only 1 partition:

Initially, the consumer commit messages manually, I changed it to auto-commit and i have the same result. #180
**Note**
We have a Kafka cluster with 4 brokers, and each one has only 2 cores(to this consumer we add a sink and source connector), is that enough?
Any help is welcome
Thank's
Samir
Contributor guide
Assessment
This issue has not been assessed yet.