Blizzard / Blizzard/node-rdkafka
Some message is missing
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
I sent one-by-one message to the producer, please see my code below,
const producer = new Kafka.Producer(
{
'bootstrap.servers': `${process.env.KAFKA_SERVER}`,
'sasl.username': `${process.env.KAFKA_USERNAME}`,
'sasl.password': `${process.env.KAFKA_PASSWORD}`,
'security.protocol': 'sasl_ssl',
'sasl.mechanisms': 'PLAIN',
'client.id': `${process.env.KAFKA_CLIENT_ID}`,
'dr_cb': true,
'enable.idempotence': true
},
{
'request.required.acks': -1,
},
);
producer.setPollInterval(100);
producer.connect({}, (err, data) => {
if (err) {
console.log('CONNECT ERROR: ', err)
}
});
producer.on('ready', async function () {
const key: string = uuid + `-` + providerRefCode + `-` + Date.now();
producer.produce(
topic,
-1,
data,
key,
Date.now()
);
producer.poll();
});
producer.on('delivery-report', function (err, report) {
if (err) {
console.log('REPORT ERROR: ', JSON.stringify(err))
}
producer.disconnect();
});
producer.on('event.error', function (err) {
console.error(err);
if (err.code == -1) {
producer.disconnect();
}
});
I found some messages missing from the target database without an error message. How to find the root cause?
Contributor guide
Research direction
Start by tracing the producer.connect, produce, poll, delivery-report, and event.error paths shown in the report, then compare delivery reports with records observed in the target database. Done means identifying a reproducible cause for the missing messages and documenting the evidence or an actionable change; the issue does not name project files or tests to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kafka, nodejs
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100