confluentinc / confluentinc/confluent-kafka-javascript

eachMessage is not concurrent when using partitionsConsumedConcurrently

Open
#223 1 comment 4 reactions 0 assignees View on GitHub
question
Dominant language
TypeScript
Stars
304
Forks
45
Avg merge
11h 47m
Merged PRs (30d)
5

Description

**Environment Information**
- OS [ubuntu 24.10]
- Node Version [20.16.0]
- confluent-kafka-javascript version [1.0.0]

**Summary**
The following code sends messages to a given topic with 2 partitions. consuming each message takes 1 second, and when using partitionsConsumedConcurrently = 2, it should consume 2 messages at a time, 1 from each partition => two messages a second.
But the code only consumes from a single partition at a time thus consuming 1 message a second.

**Steps to Reproduce**
```typescript
import { KafkaJS } from "@confluentinc/kafka-javascript";

import { config } from "../src/config";

const kafka = new KafkaJS.Kafka({
kafkaJS: {
brokers: config.kafkaBrokersAddress.split(","),
},
});

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

const topicName = "some-name";
const consumerGroup = topicName;

(async () => {
const admin = kafka.admin();
await admin.connect();

await admin.createTopics({
topics: [
{
topic: topicName,
numPartitions: 2,
},
],
});

console.log("Partitions added successfully!");
await admin.disconnect();

const producer = kafka.producer();
await producer.connect();

for (let i = 0; i < 50; i++) {
const messages = [];
for (let q = 0; q < 1000; q++) {
messages.push({ headers: {}, value: (q + i * 1000).toString() + " " + "a".repeat(1000) });
}

await producer.send({
topic: topicName,
messages: messages,
});
}
await producer.disconnect();

const consumer = kafka.consumer({
kafkaJS: {
groupId: consumerGroup,
maxWaitTimeInMs: 5000,
fromBeginning: true,
},
});
await consumer.connect();
await consumer.subscribe({ topics: [topicName] });

await consumer.run({
partitionsConsumedConcurrently: 2,
eachMessage: async ({ topic, partition, message }) => {
const value = message.value!.toString().split(" ")[0];

console.log("handling message", {
timestamp: new Date(),
partition,
topic,
value,
});

await sleep(1000);

console.log("finished message", {
timestamp: new Date(),
topic,
partition,
value,
});
},
});

await new Promise((resolve) => setTimeout(resolve, 30000));
await consumer.disconnect();
})();
```

**Output**
```
handling message {
timestamp: 2024-12-23T11:10:30.337Z,
partition: 1,
topic: 'some-name',
value: '0'
}
finished message {
timestamp: 2024-12-23T11:10:31.340Z,
topic: 'some-name',
partition: 1,
value: '0'
}
handling message {
timestamp: 2024-12-23T11:10:31.342Z,
partition: 1,
topic: 'some-name',
value: '1'
}
finished message {
timestamp: 2024-12-23T11:10:32.344Z,
topic: 'some-name',
partition: 1,
value: '1'
}
handling message {
timestamp: 2024-12-23T11:10:32.346Z,
partition: 1,
topic: 'some-name',
value: '2'
}
finished message {
timestamp: 2024-12-23T11:10:33.347Z,
topic: 'some-name',
partition: 1,
value: '2'
}
...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.