Blizzard / Blizzard/node-rdkafka

How to manually (and correctly) resolve offsets?

Open
#1,041 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.2k
Forks
403
PR merge metrics
No merged PRs in 30d

Description

So I'm wondering what is the correct way to achieve:
a) I want `node-rdkafka` to periodically auto-commit offsets
b) I want to manually store offsets to commit (in-case message processing fails)

I'm using the following to set up stream:

```typescript
const stream = KafkaConsumer.createReadStream(
{
'client.id': 'client_id',
'metadata.broker.list': 'brokers',
'enable.auto.commit': true, // Enable auto-commit
'enable.auto.offset.store': false, // We want to manually resolve offsets
'group.id': 'group-id',
'offset_commit_cb': true, // For logging purposes
'partition.assignment.strategy': 'roundrobin',
},
{
'auto.offset.reset': 'latest',
},
{
fetchSize: 100,
topics: ['topic'],
},
);
```

To process messages and resolve offsets (right after message has been successfully consumed):

```typescript
class ExampleConsumer {
public async run() {
for await (const message of this.stream) {
if (this.shouldProcessMessage(message)) {
await this.processMessage(message)
this.resolveOffset(message);
}
}
}

public resolveOffset(message: Message) {
if (this.shouldProcessMessage(message)) {
this.stream.consumer.offsetsStore([
{
offset: message.offset + 1,
partition: message.partition,
topic: message.topic,
},
]);
}
}

public shouldProcessMessage(message: Message) {
if (this.stream) {
return this.stream.consumer
.assignments()
.some((v) => v.topic === message.topic && v.partition === message.partition);
}

return false;
}
}
```

However, `consumer.offsetsStore()` method still occasionally throws `Local: Erroneous state` error....

Contributor guide

Open the contributing guide

Research direction

Start with KafkaConsumer.createReadStream and the consumer.offsetsStore() call shown in the issue, then consult the node-rdkafka and librdkafka offset-commit behavior documented for these options. Clarify the supported interaction between auto-commit, manual offset storage, assignments, and processing failures, including what conditions produce the reported error. Done means the correct usage is documented with a reproducible example or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, kafka, nodejs
Domain
distributed-systems
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.