Blizzard / Blizzard/node-rdkafka

Consumer throws `Error: Need to specify a callback`

Open
#1,004 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

**Environment Information**
- OS: MacOS 12.6.2
- Node Version (docker): 14.15.4
- Node Version (local): 14.21.2
- NPM Version (docker): 6.14.10
- NPM Version (local): 6.14.17
- C++: (not sure how to check this)
- node-rdkafka version: 2.15.0
- Docker Version: 20.10.22, build 3a2c30b
- TypeScript: 4.2.2

**Steps to Reproduce**
I have a wrapper around `consumer.consume()` method like this:
```
...
async consumeEvents(amount: number): Promise> {
const { consumer } = this;

return new Promise((resolve, reject) => {
consumer.consume(amount, (err, messages) => {
if (err) {
return reject(err);
}

if (messages.length > 0) {
return resolve(messages);
}

return resolve([]);
});
});
}
...
```
Then just call `await this.consumeEvents(100);`

**node-rdkafka Configuration Settings**
```
"client.id": "test",
"metadata.broker.list": "ip1,ip2,etc",
"group.id": "test_rdkafka",
"fetch.wait.max.ms": 50,
"socket.keepalive.enable": true,
rebalance_cb: true
```

**Additional context**
I faced a weird issue that is reproducible only in the Docker container. Above I described a wrapper that I use for consuming events in `non-flowing mode`.

### Local env (described above):
1. Run `await this.consumeEvents(100);`
2. It does connect to the broker and receives messages just fine.

### Docker env
1. Run `await this.consumeEvents(100);`
2. It throws:
```
Error: Need to specify a callback
at KafkaConsumer._consumeNum (/app/node_modules/node-rdkafka/lib/kafka-consumer.js:466:16)
at KafkaConsumer.consume (/app/node_modules/node-rdkafka/lib/kafka-consumer.js:399:10)
at /app/dist/kafka/kafka.service.js:52:22
at new Promise ()
at KafkaService.consumeEvents (/app/dist/kafka/kafka.service.js:51:16)
at KafkaService.onReady (/app/dist/kafka/kafka.service.js:35:46)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
```

[/node-rdkafka/lib/kafka-consumer.js:466:16](https://github.com/Blizzard/node-rdkafka/blob/master/lib/kafka-consumer.js#L466):
![image](https://user-images.githubusercontent.com/9624816/218253993-8458f91f-a729-40f5-a627-639fbf795adc.png)

I also found cpp [code where this error is actually being thrown, I think it's here](https://github.com/Blizzard/node-rdkafka/blob/d521e0f1f3695911dee13ba632859b071b553ab9/src/kafka-consumer.cc#L1069):
![image](https://user-images.githubusercontent.com/9624816/218254059-061dd621-38f9-4bad-9e5f-f20f3db11d07.png)

As far as I understand it validates 3rd parameter and for some reason, it's not a function but as we see from [this code it's actually a passed callback function](https://github.com/Blizzard/node-rdkafka/blob/master/lib/kafka-consumer.js#L466).

I also tried different combinations of node versions, node-rdkafka libs - nothing works in a Docker container.

Additionally, tried to build & run this code in a container on Intel-based Mac and on aws ec2 instance - the same result, just throws a described error.

Please help to investigate farther.

Contributor guide

Open the contributing guide

Research direction

Start at lib/kafka-consumer.js around KafkaConsumer.consume and _consumeNum, then inspect the linked validation in src/kafka-consumer.cc. Reproduce the callback error in the reported Docker and local Node environments using the provided wrapper and configuration. Done means identifying why the callback is rejected in Docker and documenting or correcting the differing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, kafka, nodejs, typescript
Domain
backend, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.