Automatic detection and cleanup of removed topics from consumer groups
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
Description:
Problem
Currently, when a consumer group removes topics from its subscription list, Burrow continues to track and report these removed topics as errors (stopped partitions) until either:
- The Burrow service is manually restarted, or
- The `/v3/kafka/(cluster)/consumer/(group)` DELETE endpoint is manually called
This creates operational overhead and can lead to false alerts in monitoring systems.
Current Workaround:
As mentioned in #14, the current solutions require manual intervention:
1. Restart Burrow to refresh consumer group configurations
2. Use the DELETE API endpoint to manually drop the consumer group
Proposed Enhancement:
Implement automatic detection of topic removal from consumer groups by:
1. Periodic reconciliation: Burrow should periodically compare the list of topics currently consumed by a group (from Kafka metadata) against the topics it's tracking internally
2. Topic removal detection: When a topic is no longer present in the consumer group's subscription but still exists in Burrow's internal state, automatically remove it from tracking
3. Configurable interval: Add a configuration parameter (e.g., topic-reconciliation-interval) to control how frequently this check occurs (default: 5-10 minutes)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the consumer-group tracking flow and the existing /v3/kafka/(cluster)/consumer/(group) DELETE endpoint, then inspect how Kafka metadata exposes the topics currently consumed by a group. Done means removed topics are detected and dropped from Burrow's internal tracking automatically, with a configurable reconciliation interval and documented default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100