Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka

ConsumerGroup resetting offset when topic leader changed

オープン
#663 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Java
スター
239
フォーク
59
PR マージ指標
30日以内にマージされた PR はありません

説明

I encountered a suspected bug:
when a topic is migrated from one broker to another, the consumer will detect an out of range error, and then the offset will be reset to the latest according to the auto.offset.reset:latest, resulting in a large number of messages not being actually consumed.

Here are the steps to reproduce the test environment:

1. Create a 1 replica topic and write 100,000 messages. Most of the data is uploaded to S3, and a small amount is on disk (offset:0 has been uploaded to S3)

2. Initialize a group to consume from 0, and there is a large Lag at this time is 100,000.

3. Migrate the topic from broker0 to broker1

4. Start consumer, return a error message "Fetch position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[xxx:9092 (id: 1 rack: apse1-az1)], epoch=29}} is out of range for partition topic-0, resetting offset" is displayed. At this time, the Lag becomes 0

My current solution is to set auto.offset.reset:none, but this does not seem to be a fundamental solution

kafka version:3.6.1

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。