[Clarification or Issue] Topics are being reported even if consumer has stopped consuming them
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
Scenario: Start a consumer listening to topic T as group G. Produce/Consume to build some lag information. Stop consumer and start it again with same group but different topic.
Observation: Burrow's Consumer info endpoint will always display the lag related to the old topic even when offset retention on the server has expired ( eg kafka-consumer-groups is not reporting any lag on that topic anymore). In production environments, if a topic is kept produced to then the lag will keep on growing.
Question: I know there is a Delete endpoint which seems to work by deleting the whole group. Restart Burrow will also work (as expected). I would expect/hope that Burrow would drop this topic but it does not. In tests, the topic was still there 24 hours even though the offset retention on the server was set to 3 minutes as well all the timeouts I could see in burrow being set to a few minutes at most.
is this how it is supposed to work or is this an issue with the current version?
Thanks
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Consumer info endpoint and the group/topic state it reports, then compare that behavior with kafka-consumer-groups after the consumer changes topics and offset retention expires. Review how the Delete endpoint removes group state and how a Burrow restart rebuilds it. Done means determining whether the old topic should disappear and documenting or testing the expected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- observability, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100