Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka

ConsumerGroup resetting offset when topic leader changed

Ouverte
#663 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Java
Étoiles
239
Forks
59
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

I encountered a suspected bug:
when a topic is migrated from one broker to another, the consumer will detect an out of range error, and then the offset will be reset to the latest according to the auto.offset.reset:latest, resulting in a large number of messages not being actually consumed.

Here are the steps to reproduce the test environment:

1. Create a 1 replica topic and write 100,000 messages. Most of the data is uploaded to S3, and a small amount is on disk (offset:0 has been uploaded to S3)

2. Initialize a group to consume from 0, and there is a large Lag at this time is 100,000.

3. Migrate the topic from broker0 to broker1

4. Start consumer, return a error message "Fetch position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[xxx:9092 (id: 1 rack: apse1-az1)], epoch=29}} is out of range for partition topic-0, resetting offset" is displayed. At this time, the Lag becomes 0

My current solution is to set auto.offset.reset:none, but this does not seem to be a fundamental solution

kafka version:3.6.1

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.