confluentinc / confluentinc/confluent-kafka-javascript
eachMessage is not concurrent when using partitionsConsumedConcurrently
- Dominant language
- TypeScript
- Stars
- 304
- Forks
- 45
- Avg merge
- 11h 47m
- Merged PRs (30d)
- 5
Description
**Environment Information**
- OS [ubuntu 24.10]
- Node Version [20.16.0]
- confluent-kafka-javascript version [1.0.0]
**Summary**
The following code sends messages to a given topic with 2 partitions. consuming each message takes 1 second, and when using partitionsConsumedConcurrently = 2, it should consume 2 messages at a time, 1 from each partition => two messages a second.
But the code only consumes from a single partition at a time thus consuming 1 message a second.
**Steps to Reproduce**
```typescript
import { KafkaJS } from "@confluentinc/kafka-javascript";
import { config } from "../src/config";
const kafka = new KafkaJS.Kafka({
kafkaJS: {
brokers: config.kafkaBrokersAddress.split(","),
},
});
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
const topicName = "some-name";
const consumerGroup = topicName;
(async () => {
const admin = kafka.admin();
await admin.connect();
await admin.createTopics({
topics: [
{
topic: topicName,
numPartitions: 2,
},
],
});
console.log("Partitions added successfully!");
await admin.disconnect();
const producer = kafka.producer();
await producer.connect();
for (let i = 0; i < 50; i++) {
const messages = [];
for (let q = 0; q < 1000; q++) {
messages.push({ headers: {}, value: (q + i * 1000).toString() + " " + "a".repeat(1000) });
}
await producer.send({
topic: topicName,
messages: messages,
});
}
await producer.disconnect();
const consumer = kafka.consumer({
kafkaJS: {
groupId: consumerGroup,
maxWaitTimeInMs: 5000,
fromBeginning: true,
},
});
await consumer.connect();
await consumer.subscribe({ topics: [topicName] });
await consumer.run({
partitionsConsumedConcurrently: 2,
eachMessage: async ({ topic, partition, message }) => {
const value = message.value!.toString().split(" ")[0];
console.log("handling message", {
timestamp: new Date(),
partition,
topic,
value,
});
await sleep(1000);
console.log("finished message", {
timestamp: new Date(),
topic,
partition,
value,
});
},
});
await new Promise((resolve) => setTimeout(resolve, 30000));
await consumer.disconnect();
})();
```
**Output**
```
handling message {
timestamp: 2024-12-23T11:10:30.337Z,
partition: 1,
topic: 'some-name',
value: '0'
}
finished message {
timestamp: 2024-12-23T11:10:31.340Z,
topic: 'some-name',
partition: 1,
value: '0'
}
handling message {
timestamp: 2024-12-23T11:10:31.342Z,
partition: 1,
topic: 'some-name',
value: '1'
}
finished message {
timestamp: 2024-12-23T11:10:32.344Z,
topic: 'some-name',
partition: 1,
value: '1'
}
handling message {
timestamp: 2024-12-23T11:10:32.346Z,
partition: 1,
topic: 'some-name',
value: '2'
}
finished message {
timestamp: 2024-12-23T11:10:33.347Z,
topic: 'some-name',
partition: 1,
value: '2'
}
...
```
Contributor guide
Assessment
This issue has not been assessed yet.