linkedin / linkedin/Burrow

[Clarification or Issue] Topics are being reported even if consumer has stopped consuming them

Open
#504 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Scenario: Start a consumer listening to topic T as group G. Produce/Consume to build some lag information. Stop consumer and start it again with same group but different topic.

Observation: Burrow's Consumer info endpoint will always display the lag related to the old topic even when offset retention on the server has expired ( eg kafka-consumer-groups is not reporting any lag on that topic anymore). In production environments, if a topic is kept produced to then the lag will keep on growing.

Question: I know there is a Delete endpoint which seems to work by deleting the whole group. Restart Burrow will also work (as expected). I would expect/hope that Burrow would drop this topic but it does not. In tests, the topic was still there 24 hours even though the offset retention on the server was set to 3 minutes as well all the timeouts I could see in burrow being set to a few minutes at most.

is this how it is supposed to work or is this an issue with the current version?

Thanks

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Consumer info endpoint and the group/topic state it reports, then compare that behavior with kafka-consumer-groups after the consumer changes topics and offset retention expires. Review how the Delete endpoint removes group state and how a Burrow restart rebuilds it. Done means determining whether the old topic should disappear and documenting or testing the expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
observability, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.