confluentinc / confluentinc/confluent-kafka-javascript

`KafkaConsumer.consume()` Timeout Not Respected

Open
#262 4 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
304
Forks
45
Avg merge
11h 47m
Merged PRs (30d)
5

Description

### **Description**
It appears that the `consume` method in `Kafka.KafkaConsumer` does not adhere to the specified `setDefaultConsumeTimeout`. Instead, it waits for the full batch to be filled before triggering the callback. This behaviour causes delays when consuming from low-traffic topics, as the consumer remains idle until the batch size is satisfied.

### **Expected Behaviour**
The `consume` method should respect the `setDefaultConsumeTimeout` and return the available messages within the given time frame, even if the batch is not fully populated.

### **Actual Behaviour**
The consumer only returns after the batch size (`1000` in this case) is completely filled, ignoring the timeout.

### **Code to Reproduce**
```javascript
const Kafka = require("@confluentinc/kafka-javascript");
var consumer = new Kafka.KafkaConsumer({
"metadata.broker.list": "localhost:19092",
"group.id": "confluent-kafka-javascript-consumer-flow-example",
});

const topicName = "slowtopic";
const batch = 1000;

consumer.setDefaultConsumeTimeout(100);

consumer.on("ready", function () {
console.log("Consumer ready.");
consumer.subscribe([topicName]);
consume();
});

function consume() {
if (!consumer.isConnected()) return;
console.time("Consume batch timing");
consumer.consume(batch, (err, messages) => {
if (err) {
console.error(err);
return;
}
console.log(`Received ${messages.length} messages`);
console.timeEnd("Consume batch timing");
consume();
});
}

consumer.connect();
```

```
Received 1000 messages
Consume batch timing: 10.068s
Received 1000 messages
Consume batch timing: 10.083s
Received 1000 messages
Consume batch timing: 10.074s
```

### **Environment**

- Node.js Version: `v22.4.0`
- confluent-kafka-javascript version : ` 1.2.0`
- OS: `Linux`

### **Impact**
This issue affects consumers in low-traffic environments, causing significant delays and inefficient processing during quiet periods.

### **Potential Workarounds**
Currently, need to write code to dynamically adjust the batch size based on traffic.

Many thanks
Pat

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.