Burrow not picking up some consumer groups
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
We have Kafka 1.0.0 that has been running for a few months now. We've upgrading this cluster from 0.10, to 0.11, and now to 1.0.0 in January.
When running burrow off of master, we see a list of consumer groups through the API for most of our consumers, except for a few (lets call this group: foo).
This group is shown when we run the kafka-consumer-groups admin tool to list the consumer groups, and its present in the __consumer_offsets topic. It is still not available through the burrow API.
The consumer application is built off of Kafka 1.0.0.
We are also running an older version of burrow, from September 2017, prior to the 1.0.0 release. This version of burrow DOES pickup the foo consumer group. Only when we move to a newer version of burrow, which using the .toml file, do we miss the foo consumer group.
Here is our config (with stubbed out hostnames):
pidfile="burrow.pid"
stdout-logfile="burrow.out"
access-control-allow-origin="mysite.example.com"
[logging]
filename="logs/burrow.log"
level="info"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true
[zookeeper]
servers=[ "zookeeper:2181" ]
root-path="/burrowtest"
timeout=6
[client-profile.test]
client-id="burrow-stg"
kafka-version="1.0.0"
[cluster.test-stg]
class-name="kafka"
servers=[ "stg-broker:9092" ]
client-profile="test"
topic-refresh=30
offset-refresh=5
[consumer.test-stg]
class-name="kafka"
cluster="test-stg"
servers=[ "stg-broker:9092" ]
offsets-topic="__consumer_offsets"
[cluster.test-prd]
class-name="kafka"
servers=[ "prod-broker:9092" ]
client-profile="test"
topic-refresh=30
offset-refresh=5
[consumer.test-prd]
class-name="kafka"
cluster="test-prd"
servers=[ "prod-broker:9092" ]
offsets-topic="__consumer_offsets"
[httpserver.default]
address=":3000"
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the older Burrow version with the newer TOML-based configuration and trace consumer-group discovery for Kafka 1.0.0. Reproduce the case with the supplied cluster and consumer settings, then verify that the foo group present in __consumer_offsets is also returned by the Burrow API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100