Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka

ConsumerGroup resetting offset when topic leader changed

Offen
#663 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
239
Forks
59
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

I encountered a suspected bug:
when a topic is migrated from one broker to another, the consumer will detect an out of range error, and then the offset will be reset to the latest according to the auto.offset.reset:latest, resulting in a large number of messages not being actually consumed.

Here are the steps to reproduce the test environment:

1. Create a 1 replica topic and write 100,000 messages. Most of the data is uploaded to S3, and a small amount is on disk (offset:0 has been uploaded to S3)

2. Initialize a group to consume from 0, and there is a large Lag at this time is 100,000.

3. Migrate the topic from broker0 to broker1

4. Start consumer, return a error message "Fetch position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[xxx:9092 (id: 1 rack: apse1-az1)], epoch=29}} is out of range for partition topic-0, resetting offset" is displayed. At this time, the Lag becomes 0

My current solution is to set auto.offset.reset:none, but this does not seem to be a fundamental solution

kafka version:3.6.1

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.