Blizzard / Blizzard/node-rdkafka

Node-rdkafka creates hundreds of threads

Open
#947 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

**Environment Information**
- OS [e.g. Mac, Arch, Windows 10]: Debian Stretch (Docker)
- Node Version [e.g. 8.2.1]: 8.3.1
- NPM Version [e.g. 5.4.2]:
- C++ Toolchain [e.g. Visual Studio, llvm, g++]:
- node-rdkafka version [e.g. 2.3.3]: ^2.12.0

**node-rdkafka Configuration Settings**

`this.kafkaConsumer = new KafkaConsumer(
{
'group.id': randomUUID(),
'metadata.broker.list': this.brokers.join(','),
'enable.auto.commit': false,
},
{
'auto.offset.reset': 'latest'
},
);
`
`this.kafkaConsumer.subscribe([this.subscription.topic]);`
`this.kafkaConsumer.consume(maxInFlight, this.consumeCallback);`

**Additional context**
I have created about 20 consumers with noderdkafka, each subscribed to one topic.
When I start the Programm all Consumers works as expected, but then rdKafka creates more and more threads until other program processes are blocked. After a while, rdKafka created >400 threads for 20 consumers. These 400 threads also existing even if no messages being delivered for some time.

**Is this normal behavior of the library or was something implemented incorrectly?**
**Is it possible to run a few hundred consumers on one machine?**
Screen Shot 2022-04-12 at 09 39 40

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported setup with 20 KafkaConsumer instances, one topic subscription per consumer, and the shown consume call. Measure thread counts over time while messages are and are not being delivered. Done means establishing whether the thread growth is expected or indicates an implementation problem, with the relevant cause and a clear resolution identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.