Offset increase by 2 due to control batchesc
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
Using a transactional producer, it is not guaranteed that offsets are always incremented by 1. Due to control batches, it is possible that offsets are incremented by 2.
This can lead to false positives, as the lag can be equal to 1, although the consumer has consumed all messages available.
Has anyone come across this and how did you reduce the number of false positives?
Resources:
Control Batch: http://kafka.apache.org/documentation/#controlbatch
Issue observed: https://stackoverflow.com/questions/56182606/in-kafka-when-producing-message-with-transactional-consumer-offset-doubled-upc
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Burrow's consumer-lag calculation and compare its offset assumptions with Kafka's Control Batch documentation linked in the issue. Reproduce the transactional-producer case described in the issue and determine how lag equal to 1 is currently classified; done means the resulting lag check no longer reports a false positive for consumed messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100