Blizzard / Blizzard/node-rdkafka
Consumer stop messages consuming from specific topic after some time of running
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
**Environment Information**
- OS : docker image based on node:12.13.1-alpine
- Node Version : 12.13.1
- node-rdkafka version : latest
The below code snippet is working fine. But sometimes it's stopping reading messages from **specific** Kafka's topic (we are having about 20 topics with same pattern). We are not getting any errors. After service restart consuming continue as usual.
I need your assistance to resolve the above issue
```
import { ConsumerStream, createReadStream } from 'node-rdkafka';
const kafkaConsumer = createConsumerStream(shutdown, config.kafka.topics);
kafkaConsumer.on('data', async (rawMessage) => {
const {
topic, partition, offset, value
} = rawMessage;
try {
await processKafkaMessage(rawMessage);
kafkaConsumer.consumer.commit({
topic: topic,
partition: partition,
offset: offset + 1
});
} catch (err) {
logger.error('Failed to process inbound kafka message');
}
});
export const createConsumerStream = (shutdown, topics:Array):ConsumerStream => {
const globalConfig = {
'metadata.broker.list': [
/^myservice\.[^.]*\.errors/,
],
'group.id': 'my_group_1',
'enable.auto.commit': false,
'partition.assignment.strategy': 'roundrobin',
'topic.metadata.refresh.interval.ms': 30 * 100,
'batch.num.messages': 100000,
'queued.max.messages.kbytes': 10000,
'fetch.message.max.bytes': 10000,
'fetch.max.bytes': 524288000,
'retry.backoff.ms': 200,
retries: 5
};
const topicConfig = { 'auto.offset.reset': 'earliest' };
const streamOptions = {
topics: topics,
waitInterval: batchMaxTime,
fetchSize: batchMaxSize
};
const stream:ConsumerStream = createReadStream(globalConfig, topicConfig, streamOptions);
stream.on('error', (err) => {
logger.error('Error in kafka consumer stream', {
error_msg: err.message,
error_name: err.name
});
});
stream.consumer.on('event.error', (err) => {
if (err.stack === 'Error: Local: Broker transport failure') return;
logger.error('Error in kafka consumer');
stream.emit('rd-kafka-error', err);
});
stream.consumer.on('rebalance', ({ message }, assignment) => {
logger.info('Rebalance event', { assigned_topics: assignment });
});
return stream;
};
```
Contributor guide
Research direction
Start with createConsumerStream and createReadStream, then trace the data, error, event.error, and rebalance handlers in the provided example. Reproduce the silent stop for one topic while capturing the configured consumer events and assignments. Done means the cause is identified and a verified fix prevents the topic from requiring a service restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js
- Domain
- distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100