confluentinc / confluentinc/confluent-kafka-javascript

Consumption rate of messages from one topic effects the rate of consumption from another topic

Open
#247 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
304
Forks
45
Avg merge
11h 47m
Merged PRs (30d)
5

Description

**Environment Information**
- OS [ubuntu 24.10]:
- Node Version [20.16.0]:
- confluent-kafka-javascript version [1.0.0]:

**Summary**

The following code produces messages to 2 topics and then consumes from the 2 topics with `partitionsConsumedConcurrently:2` . When consuming using eachBatch, only entries from the "slow topic" are consumed, blocking the consumption of the "fast topic". In other executions, some of the "fast topic" messages are consumed but are still later blocked by the consumption of "slow topic" messages.

The expected behavior would be to consume all the "fast topic" messages very quickly, and for the existence of another messages in another topic not effect the consumption rate of the fast topic.

**Reproduce**

```typescript
import { KafkaJS } from "@confluentinc/kafka-javascript";

import { config } from "../src/config";

const kafka = new KafkaJS.Kafka({
kafkaJS: {
brokers: config.kafkaBrokersAddress.split(","),
},
});

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

const fastTopicName = "fast-topic5";
const slowTopicName = "slow-topic5";
const consumerGroup = "cg5";

(async () => {
const admin = kafka.admin();
await admin.connect();

await admin.createTopics({
topics: [
{
topic: fastTopicName,
numPartitions: 1,
},
{
topic: slowTopicName,
numPartitions: 1,
},
],
});

console.log("Partitions added successfully!");
await admin.disconnect();

const producer = kafka.producer();
await producer.connect();

for (let i = 0; i < 5; i++) {
const messages = [];
for (let q = 0; q < 50; q++) {
messages.push({ headers: {}, value: (q + i * 1000).toString() + " " + "a".repeat(1000) });
}

await producer.send({
topic: slowTopicName,
messages: messages,
});

await producer.send({
topic: fastTopicName,
messages: messages,
});
}
await producer.disconnect();

const consumer = kafka.consumer({
kafkaJS: {
groupId: consumerGroup,
maxWaitTimeInMs: 5000,
fromBeginning: true,
},
});
await consumer.connect();
await consumer.subscribe({ topics: [slowTopicName, fastTopicName] });

const messagesConsumed: Record = { [fastTopicName]: 0, [slowTopicName]: 0 };
await consumer.run({
partitionsConsumedConcurrently: 2,
eachBatch: async ({ batch }) => {
console.log("handling batch", {
partition: batch.partition,
topic: batch.topic,
});

if (batch.topic === slowTopicName) {
await sleep(500);
}

messagesConsumed[batch.topic] += batch.messages.length;

console.log("finished batch", {
time: new Date(),
partition: batch.partition,
consumed: JSON.stringify(messagesConsumed),
});
},
});

await new Promise((resolve) => setTimeout(resolve, 30000));
await consumer.disconnect();
})();
```

**Output**

```
handling batch { partition: 0, topic: 'slow-topic1' }
finished batch {
time: 2025-01-22T17:00:55.045Z,
partition: 0,
consumed: '{"fast-topic1":0,"slow-topic1":1}'
}
handling batch { partition: 0, topic: 'slow-topic1' }
finished batch {
time: 2025-01-22T17:00:55.546Z,
partition: 0,
consumed: '{"fast-topic1":0,"slow-topic1":2}'
}
handling batch { partition: 0, topic: 'slow-topic1' }
finished batch {
time: 2025-01-22T17:00:56.048Z,
partition: 0,
consumed: '{"fast-topic1":0,"slow-topic1":4}'
}
...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.