Blizzard / Blizzard/node-rdkafka
Node-rdkafka creates hundreds of threads
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
**Environment Information**
- OS [e.g. Mac, Arch, Windows 10]: Debian Stretch (Docker)
- Node Version [e.g. 8.2.1]: 8.3.1
- NPM Version [e.g. 5.4.2]:
- C++ Toolchain [e.g. Visual Studio, llvm, g++]:
- node-rdkafka version [e.g. 2.3.3]: ^2.12.0
**node-rdkafka Configuration Settings**
`this.kafkaConsumer = new KafkaConsumer(
{
'group.id': randomUUID(),
'metadata.broker.list': this.brokers.join(','),
'enable.auto.commit': false,
},
{
'auto.offset.reset': 'latest'
},
);
`
`this.kafkaConsumer.subscribe([this.subscription.topic]);`
`this.kafkaConsumer.consume(maxInFlight, this.consumeCallback);`
**Additional context**
I have created about 20 consumers with noderdkafka, each subscribed to one topic.
When I start the Programm all Consumers works as expected, but then rdKafka creates more and more threads until other program processes are blocked. After a while, rdKafka created >400 threads for 20 consumers. These 400 threads also existing even if no messages being delivered for some time.
**Is this normal behavior of the library or was something implemented incorrectly?**
**Is it possible to run a few hundred consumers on one machine?**

Contributor guide
Research direction
Start by reproducing the reported setup with 20 KafkaConsumer instances, one topic subscription per consumer, and the shown consume call. Measure thread counts over time while messages are and are not being delivered. Done means establishing whether the thread growth is expected or indicates an implementation problem, with the relevant cause and a clear resolution identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100