confluentinc / confluentinc/confluent-kafka-javascript

uncommittedOffsets : Clarification and Alternatives

Open
#351 2 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
TypeScript
Stars
304
Forks
45
Avg merge
11h 47m
Merged PRs (30d)
5

Description

Hi Team,

We are migrating to Javascript client from KafkaJS and using APIs under:
`const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS;`

We would like to get clarification on uncommittedOffsets support in the migration documentation present here:
**Within the eachBatch callback, use of uncommittedOffsets is unsupported**
[Link to documentation](https://docs.confluent.io/kafka-clients/javascript/current/migration.html#javascript-migrate-from-kafkajs:~:text=Within%20the%20eachBatch%20callback%2C%20use%20of%20uncommittedOffsets%20is%20unsupported%2C)

Is there any alternative provided to track and commit offsets with a similar fine-tuned control? I am attaching some example usages below.

Example Usage 1:
```
await resolveOffsets(batch.messages, resolveOffset)
await commitOffsetsIfNecessary(uncommittedOffsets())
```

Usage 2: Where we track and update uncommittedOffsets

```
const uncommittedOffsets = []
....
consumer.uncommittedOffsets = consumer.uncommittedOffsets.concat(uncommittedOffsets)
....
if (consumer.uncommittedOffsets.length === 0) {
return done(null, false)
}
consumer.commitOffsetsIfNecessary(consumer.uncommittedOffsets)
.then(() => {
consumer.uncommittedOffsets = []
return done(null, true)
})
.catch(err => {
return done(err, false)
})
}
```

We do not want to use `autocommit :true` since, it does not provide the finer-grained control over when and how offsets are committed. So, please let us know if there's any alternative.

Thanks in advance.

Contributor guide

Open the contributing guide

Research direction

Start with the migration documentation section on uncommittedOffsets and compare the eachBatch examples with the two KafkaJS usage patterns in the issue. Determine whether the client exposes a supported equivalent for fine-grained offset tracking and committing; done means the migration guidance clearly answers the question or identifies the supported limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
stream-processing
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.