linkedin / linkedin/Burrow

Consumers are not getting listed in burrow

Open
#578 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

I have a dockerized burrow setup to monitor my dev kafka environment. When I check **/v3/kafka/kafka-dev/consumer**, it lists only 2 consumers.

----
"consumers": [
"adept-tracker-atkins",
"burrow-kafka-dev"
],
"request": {
"url": "/v3/kafka/kafka-dev/consumer",
"host": "a9720a5c007a"
}
}
----

But I have 15 consumers in total. All the consumers are committing offsets to kafka. When I restart the burrow container it lists all the consumers but their status will be "NOT FOUND".

----

{"error":false,"message":"consumer list returned","consumers":["console-consumer-22451","adept-egress-processing-generic2","console-consumer-83905","adept-tracker-atkins-dev","connect-ingress-gps-location-s3","connect-egress-gps-location-s3","console-consumer-92438","burrow-kafka-dev","connect-driving-events-egress-log-s3","connect-batch-reject-s3","connect-egress-gps-location-log-s3","connect-reject-gps-location-s3","adept-tracker-atkins","connect-reject-ingress-gps-location-s3","console-consumer-24933","adept-egress-processing-generic1","connect-driving-events-validated-enriched-s3","adept-egress-processing-tomtom","connect-driving-events-ingress-raw-rejects-s3","connect-driving-events-ingress-raw-s3","adept-egress-processing-example","adept-stream-core-processing-drivingevents"],"request":{"url":"/v3/kafka/kafka-dev/consumer","host":"e0275992bad7"}}
----

My burrow.toml file is as follows.

----
[general]
pidfile="burrow.pid"
stdout-logfile="burrow.out"

[logging]
filename="logs/burrow.log"
level="info"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true

[zookeeper]
servers=[ "zookeeper:2181" ]
timeout=6
root-path="/burrow"

[client-profile.kafka10]
kafka-version="0.10.1.0"
client-id="burrow-client"

[client-profile.zk-kafka10]
kafka-version="0.10.1.0"
client-id="burrow-client"

[cluster.kafka-dev]
class-name="kafka"
servers=[ "kafka:9092" ]
client-profile="kafka10"
topic-refresh=60
offset-refresh=30

[consumer.kafka-dev]
class-name="kafka"
cluster="kafka-dev"
client-profile="kafka10"
servers=[ "kafka:9092" ]
group-blacklist="^(console-consumer-|python-kafka-consumer-).*$"
group-whitelist=""

[consumer.kafka-dev_zk]
class-name="kafka_zk"
cluster="kafka-dev"
client-profile="zk-kafka10"
servers=[ "zookeeper:2181" ]
zookeeper-timeout=30
group-blacklist="^(console-consumer-|python-kafka-consumer-).*$"
group-whitelist=""

[httpserver.default]
address=":8005"

[storage.default]
class-name="inmemory"
workers=20
intervals=15
expire-group=604800
min-distance=1
----

@toddpalino Can you help me here to get this working. I have to go to production soon.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the /v3/kafka/kafka-dev/consumer endpoint and the consumer.kafka-dev and consumer.kafka-dev_zk sections in burrow.toml. Reproduce the restart behavior and trace how consumer groups are discovered and assigned status; done means all committing consumers are listed without incorrectly showing them as NOT FOUND.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
observability, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.