linkedin / linkedin/Burrow

Automatic detection and cleanup of removed topics from consumer groups

Open
#854 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Description:
Problem
Currently, when a consumer group removes topics from its subscription list, Burrow continues to track and report these removed topics as errors (stopped partitions) until either:

- The Burrow service is manually restarted, or
- The `/v3/kafka/(cluster)/consumer/(group)` DELETE endpoint is manually called

This creates operational overhead and can lead to false alerts in monitoring systems.

Current Workaround:
As mentioned in #14, the current solutions require manual intervention:
1. Restart Burrow to refresh consumer group configurations
2. Use the DELETE API endpoint to manually drop the consumer group

Proposed Enhancement:
Implement automatic detection of topic removal from consumer groups by:
1. Periodic reconciliation: Burrow should periodically compare the list of topics currently consumed by a group (from Kafka metadata) against the topics it's tracking internally
2. Topic removal detection: When a topic is no longer present in the consumer group's subscription but still exists in Burrow's internal state, automatically remove it from tracking
3. Configurable interval: Add a configuration parameter (e.g., topic-reconciliation-interval) to control how frequently this check occurs (default: 5-10 minutes)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the consumer-group tracking flow and the existing /v3/kafka/(cluster)/consumer/(group) DELETE endpoint, then inspect how Kafka metadata exposes the topics currently consumed by a group. Done means removed topics are detected and dropped from Burrow's internal tracking automatically, with a configurable reconciliation interval and documented default.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.