Blizzard / Blizzard/node-rdkafka
Consumer throws `Error: Need to specify a callback`
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
**Environment Information**
- OS: MacOS 12.6.2
- Node Version (docker): 14.15.4
- Node Version (local): 14.21.2
- NPM Version (docker): 6.14.10
- NPM Version (local): 6.14.17
- C++: (not sure how to check this)
- node-rdkafka version: 2.15.0
- Docker Version: 20.10.22, build 3a2c30b
- TypeScript: 4.2.2
**Steps to Reproduce**
I have a wrapper around `consumer.consume()` method like this:
```
...
async consumeEvents(amount: number): Promise> {
const { consumer } = this;
return new Promise((resolve, reject) => {
consumer.consume(amount, (err, messages) => {
if (err) {
return reject(err);
}
if (messages.length > 0) {
return resolve(messages);
}
return resolve([]);
});
});
}
...
```
Then just call `await this.consumeEvents(100);`
**node-rdkafka Configuration Settings**
```
"client.id": "test",
"metadata.broker.list": "ip1,ip2,etc",
"group.id": "test_rdkafka",
"fetch.wait.max.ms": 50,
"socket.keepalive.enable": true,
rebalance_cb: true
```
**Additional context**
I faced a weird issue that is reproducible only in the Docker container. Above I described a wrapper that I use for consuming events in `non-flowing mode`.
### Local env (described above):
1. Run `await this.consumeEvents(100);`
2. It does connect to the broker and receives messages just fine.
### Docker env
1. Run `await this.consumeEvents(100);`
2. It throws:
```
Error: Need to specify a callback
at KafkaConsumer._consumeNum (/app/node_modules/node-rdkafka/lib/kafka-consumer.js:466:16)
at KafkaConsumer.consume (/app/node_modules/node-rdkafka/lib/kafka-consumer.js:399:10)
at /app/dist/kafka/kafka.service.js:52:22
at new Promise ()
at KafkaService.consumeEvents (/app/dist/kafka/kafka.service.js:51:16)
at KafkaService.onReady (/app/dist/kafka/kafka.service.js:35:46)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
```
[/node-rdkafka/lib/kafka-consumer.js:466:16](https://github.com/Blizzard/node-rdkafka/blob/master/lib/kafka-consumer.js#L466):

I also found cpp [code where this error is actually being thrown, I think it's here](https://github.com/Blizzard/node-rdkafka/blob/d521e0f1f3695911dee13ba632859b071b553ab9/src/kafka-consumer.cc#L1069):

As far as I understand it validates 3rd parameter and for some reason, it's not a function but as we see from [this code it's actually a passed callback function](https://github.com/Blizzard/node-rdkafka/blob/master/lib/kafka-consumer.js#L466).
I also tried different combinations of node versions, node-rdkafka libs - nothing works in a Docker container.
Additionally, tried to build & run this code in a container on Intel-based Mac and on aws ec2 instance - the same result, just throws a described error.
Please help to investigate farther.
Contributor guide
Research direction
Start at lib/kafka-consumer.js around KafkaConsumer.consume and _consumeNum, then inspect the linked validation in src/kafka-consumer.cc. Reproduce the callback error in the reported Docker and local Node environments using the provided wrapper and configuration. Done means identifying why the callback is rejected in Docker and documenting or correcting the differing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kafka, nodejs, typescript
- Domain
- backend, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100