Blizzard / Blizzard/node-rdkafka

The consumer does not consume all partitions fairly

Open
#808 2 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

**Environment Information**
- OS [CentOS Linux]:
- Node Version [12 and 14]:
- NPM Version [6.14.5]:
- node-rdkafka version [2.9.0]:

**node-rdkafka Configuration Settings**
I use non-following mode consumer and each batch contains 2000 records, consumer configuration:

```
{
groupId: 'frg',
bootstrapServer: '10.120.1.20:9092',
topic: 'rdo-stocks-v1',
clientId: 'rdo-stock-quantity',
enableAutoCommit: true,
enableAutoOffsetStore: true,
socketKeepaliveEnable: 'true',
securityProtocol: 'SASL_PLAINTEXT',
saslMechanisms: 'SCRAM-SHA-512',
saslUsername: 'rdUser',
saslPassword: 'secret',
consumerMaxPollInterval: '200000',
consumerAutoOffsetReset: 'earliest',
consumerCommitInterval: 500,
consumerPartitionAssignmentStrategy: 'roundrobin',
debug: 'fetch',
consumerHeartbeatInternalMs: 10000,
consumerSessionTimeout: 20000,
consumerFetchMessageSizeByte: 1048576,
messageMaxBytes: 1548576,
consumerQueuedMaxMessagesKbytes: 1048576,
consumerMaxPollRecords: 2000,
consumerDefaultTimeout: 100
}
```
UV_THREADPOOL_SIZE is set to 30.
The consumer run on in a docker container and each consumer runs on a different host (8 cores).

**Additional context**

The consumer consumes messages well, but as we can see in the picture below, each Kafka broker is the leader of two partitions and the consumer consumes only 1 partition:

![image](https://user-images.githubusercontent.com/12759390/84998685-7106ea00-b150-11ea-906f-9ed5167ac5ab.png)

Initially, the consumer commit messages manually, I changed it to auto-commit and i have the same result. #180
**Note**
We have a Kafka cluster with 4 brokers, and each one has only 2 cores(to this consumer we add a sink and source connector), is that enough?

Any help is welcome

Thank's
Samir

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.