Consumers are not getting listed in burrow
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
I have a dockerized burrow setup to monitor my dev kafka environment. When I check **/v3/kafka/kafka-dev/consumer**, it lists only 2 consumers.
----
"consumers": [
"adept-tracker-atkins",
"burrow-kafka-dev"
],
"request": {
"url": "/v3/kafka/kafka-dev/consumer",
"host": "a9720a5c007a"
}
}
----
But I have 15 consumers in total. All the consumers are committing offsets to kafka. When I restart the burrow container it lists all the consumers but their status will be "NOT FOUND".
----
{"error":false,"message":"consumer list returned","consumers":["console-consumer-22451","adept-egress-processing-generic2","console-consumer-83905","adept-tracker-atkins-dev","connect-ingress-gps-location-s3","connect-egress-gps-location-s3","console-consumer-92438","burrow-kafka-dev","connect-driving-events-egress-log-s3","connect-batch-reject-s3","connect-egress-gps-location-log-s3","connect-reject-gps-location-s3","adept-tracker-atkins","connect-reject-ingress-gps-location-s3","console-consumer-24933","adept-egress-processing-generic1","connect-driving-events-validated-enriched-s3","adept-egress-processing-tomtom","connect-driving-events-ingress-raw-rejects-s3","connect-driving-events-ingress-raw-s3","adept-egress-processing-example","adept-stream-core-processing-drivingevents"],"request":{"url":"/v3/kafka/kafka-dev/consumer","host":"e0275992bad7"}}
----
My burrow.toml file is as follows.
----
[general]
pidfile="burrow.pid"
stdout-logfile="burrow.out"
[logging]
filename="logs/burrow.log"
level="info"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true
[zookeeper]
servers=[ "zookeeper:2181" ]
timeout=6
root-path="/burrow"
[client-profile.kafka10]
kafka-version="0.10.1.0"
client-id="burrow-client"
[client-profile.zk-kafka10]
kafka-version="0.10.1.0"
client-id="burrow-client"
[cluster.kafka-dev]
class-name="kafka"
servers=[ "kafka:9092" ]
client-profile="kafka10"
topic-refresh=60
offset-refresh=30
[consumer.kafka-dev]
class-name="kafka"
cluster="kafka-dev"
client-profile="kafka10"
servers=[ "kafka:9092" ]
group-blacklist="^(console-consumer-|python-kafka-consumer-).*$"
group-whitelist=""
[consumer.kafka-dev_zk]
class-name="kafka_zk"
cluster="kafka-dev"
client-profile="zk-kafka10"
servers=[ "zookeeper:2181" ]
zookeeper-timeout=30
group-blacklist="^(console-consumer-|python-kafka-consumer-).*$"
group-whitelist=""
[httpserver.default]
address=":8005"
[storage.default]
class-name="inmemory"
workers=20
intervals=15
expire-group=604800
min-distance=1
----
@toddpalino Can you help me here to get this working. I have to go to production soon.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the /v3/kafka/kafka-dev/consumer endpoint and the consumer.kafka-dev and consumer.kafka-dev_zk sections in burrow.toml. Reproduce the restart behavior and trace how consumer groups are discovered and assigned status; done means all committing consumers are listed without incorrectly showing them as NOT FOUND.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- observability, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100