Blizzard / Blizzard/node-rdkafka

Some message is missing

Open
#1,046 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

I sent one-by-one message to the producer, please see my code below,

const producer = new Kafka.Producer(
{
'bootstrap.servers': `${process.env.KAFKA_SERVER}`,
'sasl.username': `${process.env.KAFKA_USERNAME}`,
'sasl.password': `${process.env.KAFKA_PASSWORD}`,
'security.protocol': 'sasl_ssl',
'sasl.mechanisms': 'PLAIN',
'client.id': `${process.env.KAFKA_CLIENT_ID}`,
'dr_cb': true,
'enable.idempotence': true
},
{
'request.required.acks': -1,
},
);

producer.setPollInterval(100);

producer.connect({}, (err, data) => {
if (err) {
console.log('CONNECT ERROR: ', err)
}
});

producer.on('ready', async function () {
const key: string = uuid + `-` + providerRefCode + `-` + Date.now();
producer.produce(
topic,
-1,
data,
key,
Date.now()
);
producer.poll();
});

producer.on('delivery-report', function (err, report) {
if (err) {
console.log('REPORT ERROR: ', JSON.stringify(err))
}
producer.disconnect();
});

producer.on('event.error', function (err) {
console.error(err);
if (err.code == -1) {
producer.disconnect();
}
});

I found some messages missing from the target database without an error message. How to find the root cause?

Contributor guide

Open the contributing guide

Research direction

Start by tracing the producer.connect, produce, poll, delivery-report, and event.error paths shown in the report, then compare delivery reports with records observed in the target database. Done means identifying a reproducible cause for the missing messages and documenting the evidence or an actionable change; the issue does not name project files or tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, kafka, nodejs
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.