linkedin / linkedin/Burrow

Offset increase by 2 due to control batchesc

Open
#651 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Using a transactional producer, it is not guaranteed that offsets are always incremented by 1. Due to control batches, it is possible that offsets are incremented by 2.

This can lead to false positives, as the lag can be equal to 1, although the consumer has consumed all messages available.

Has anyone come across this and how did you reduce the number of false positives?

Resources:
Control Batch: http://kafka.apache.org/documentation/#controlbatch
Issue observed: https://stackoverflow.com/questions/56182606/in-kafka-when-producing-message-with-transactional-consumer-offset-doubled-upc

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Burrow's consumer-lag calculation and compare its offset assumptions with Kafka's Control Batch documentation linked in the issue. Reproduce the transactional-producer case described in the issue and determine how lag equal to 1 is currently classified; done means the resulting lag check no longer reports a false positive for consumed messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.