linkedin / linkedin/Burrow

Burrow not picking up some consumer groups

Open
#377 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

We have Kafka 1.0.0 that has been running for a few months now. We've upgrading this cluster from 0.10, to 0.11, and now to 1.0.0 in January.

When running burrow off of master, we see a list of consumer groups through the API for most of our consumers, except for a few (lets call this group: foo).

This group is shown when we run the kafka-consumer-groups admin tool to list the consumer groups, and its present in the __consumer_offsets topic. It is still not available through the burrow API.

The consumer application is built off of Kafka 1.0.0.

We are also running an older version of burrow, from September 2017, prior to the 1.0.0 release. This version of burrow DOES pickup the foo consumer group. Only when we move to a newer version of burrow, which using the .toml file, do we miss the foo consumer group.

Here is our config (with stubbed out hostnames):

pidfile="burrow.pid"
stdout-logfile="burrow.out"
access-control-allow-origin="mysite.example.com"

[logging]
filename="logs/burrow.log"
level="info"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true

[zookeeper]
servers=[ "zookeeper:2181" ]
root-path="/burrowtest"
timeout=6

[client-profile.test]
client-id="burrow-stg"
kafka-version="1.0.0"

[cluster.test-stg]
class-name="kafka"
servers=[ "stg-broker:9092" ]
client-profile="test"
topic-refresh=30
offset-refresh=5

[consumer.test-stg]
class-name="kafka"
cluster="test-stg"
servers=[ "stg-broker:9092" ]
offsets-topic="__consumer_offsets"

[cluster.test-prd]
class-name="kafka"
servers=[ "prod-broker:9092" ]
client-profile="test"
topic-refresh=30
offset-refresh=5

[consumer.test-prd]
class-name="kafka"
cluster="test-prd"
servers=[ "prod-broker:9092" ]
offsets-topic="__consumer_offsets"


[httpserver.default]
address=":3000"

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the older Burrow version with the newer TOML-based configuration and trace consumer-group discovery for Kafka 1.0.0. Reproduce the case with the supplied cluster and consumer settings, then verify that the foo group present in __consumer_offsets is also returned by the Burrow API.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.