Blizzard / Blizzard/node-rdkafka
How to manually (and correctly) resolve offsets?
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
So I'm wondering what is the correct way to achieve:
a) I want `node-rdkafka` to periodically auto-commit offsets
b) I want to manually store offsets to commit (in-case message processing fails)
I'm using the following to set up stream:
```typescript
const stream = KafkaConsumer.createReadStream(
{
'client.id': 'client_id',
'metadata.broker.list': 'brokers',
'enable.auto.commit': true, // Enable auto-commit
'enable.auto.offset.store': false, // We want to manually resolve offsets
'group.id': 'group-id',
'offset_commit_cb': true, // For logging purposes
'partition.assignment.strategy': 'roundrobin',
},
{
'auto.offset.reset': 'latest',
},
{
fetchSize: 100,
topics: ['topic'],
},
);
```
To process messages and resolve offsets (right after message has been successfully consumed):
```typescript
class ExampleConsumer {
public async run() {
for await (const message of this.stream) {
if (this.shouldProcessMessage(message)) {
await this.processMessage(message)
this.resolveOffset(message);
}
}
}
public resolveOffset(message: Message) {
if (this.shouldProcessMessage(message)) {
this.stream.consumer.offsetsStore([
{
offset: message.offset + 1,
partition: message.partition,
topic: message.topic,
},
]);
}
}
public shouldProcessMessage(message: Message) {
if (this.stream) {
return this.stream.consumer
.assignments()
.some((v) => v.topic === message.topic && v.partition === message.partition);
}
return false;
}
}
```
However, `consumer.offsetsStore()` method still occasionally throws `Local: Erroneous state` error....
Contributor guide
Research direction
Start with KafkaConsumer.createReadStream and the consumer.offsetsStore() call shown in the issue, then consult the node-rdkafka and librdkafka offset-commit behavior documented for these options. Clarify the supported interaction between auto-commit, manual offset storage, assignments, and processing failures, including what conditions produce the reported error. Done means the correct usage is documented with a reproducible example or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kafka, nodejs
- Domain
- distributed-systems
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100