Blizzard / Blizzard/node-rdkafka

Consumer stop messages consuming from specific topic after some time of running

Open
#931 1 comment 8 reactions 0 assignees View on GitHub
stale
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

**Environment Information**
- OS : docker image based on node:12.13.1-alpine
- Node Version : 12.13.1
- node-rdkafka version : latest

The below code snippet is working fine. But sometimes it's stopping reading messages from **specific** Kafka's topic (we are having about 20 topics with same pattern). We are not getting any errors. After service restart consuming continue as usual.

I need your assistance to resolve the above issue

```
import { ConsumerStream, createReadStream } from 'node-rdkafka';

const kafkaConsumer = createConsumerStream(shutdown, config.kafka.topics);
kafkaConsumer.on('data', async (rawMessage) => {
const {
topic, partition, offset, value
} = rawMessage;

try {
await processKafkaMessage(rawMessage);
kafkaConsumer.consumer.commit({
topic: topic,
partition: partition,
offset: offset + 1
});
} catch (err) {
logger.error('Failed to process inbound kafka message');
}
});

export const createConsumerStream = (shutdown, topics:Array):ConsumerStream => {

const globalConfig = {
'metadata.broker.list': [
/^myservice\.[^.]*\.errors/,
],
'group.id': 'my_group_1',
'enable.auto.commit': false,
'partition.assignment.strategy': 'roundrobin',
'topic.metadata.refresh.interval.ms': 30 * 100,
'batch.num.messages': 100000,
'queued.max.messages.kbytes': 10000,
'fetch.message.max.bytes': 10000,
'fetch.max.bytes': 524288000,
'retry.backoff.ms': 200,
retries: 5
};

const topicConfig = { 'auto.offset.reset': 'earliest' };

const streamOptions = {
topics: topics,
waitInterval: batchMaxTime,
fetchSize: batchMaxSize
};

const stream:ConsumerStream = createReadStream(globalConfig, topicConfig, streamOptions);

stream.on('error', (err) => {
logger.error('Error in kafka consumer stream', {
error_msg: err.message,
error_name: err.name
});
});

stream.consumer.on('event.error', (err) => {
if (err.stack === 'Error: Local: Broker transport failure') return;

logger.error('Error in kafka consumer');

stream.emit('rd-kafka-error', err);
});

stream.consumer.on('rebalance', ({ message }, assignment) => {
logger.info('Rebalance event', { assigned_topics: assignment });
});

return stream;
};

```

Contributor guide

Open the contributing guide

Research direction

Start with createConsumerStream and createReadStream, then trace the data, error, event.error, and rebalance handlers in the provided example. Reproduce the silent stop for one topic while capturing the configured consumer events and assignments. Done means the cause is identified and a verified fix prevents the topic from requiring a service restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.