linkedin / linkedin/Burrow

Burrow does not seem to work with offsets stored in Zookeeper

Open
#544 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Hi everybody,

I have these Kafka brokers writing offsets to a Zookeeper cluster.
Here is burrow.toml:
```
[SRV_MONRTDM@pmonigovas01v ~]$ cat burrow30/burrow.toml
[general]
access-control-allow-origin="*" #Header value to put in http response

[zookeeper]
servers=["10.207.180.133:2181", "10.207.180.134:2181", "10.207.180.135:2181" ]
timeout=6 #expiration timeout for zookeeper sessions.
root-path="/burrow" #full path to zookeeper node that burrow will be allowed to write into.

[client-profile.myclient] #this client profile name is myclient
kafka-version="0.10" #kafka server version
client-id="burrow-client" # a string to be passed to kafka as client Id

[httpserver.mylistener]
address=":8000" #port to send http request

[storage.mystorage]
class-name="inmemory"
intervals=10 #no. of offsets to store for each partition
expire-group=604800 # no. of seconds after which a group will be purged if it has not commited offset

[cluster.hdp30]
class-name="kafka"
servers=[ "10.207.180.142:6667", "10.207.180.143:6667", "10.207.180.144:6667", "10.207.180.145:6667" ]
client-profile="myclient" #profile name defined as in client profile subheading.
topic-refresh=10
offset-refresh=10

[consumer.hdp-burrow30]
class-name="kafka_zk"
servers=["10.207.180.133:2181", "10.207.180.134:2181", "10.207.180.135:2181" ]
cluster="hdp30" # sub heading name defined in above cluster configuration
zookeeper-timeout=5
zookeeper-path=""
```

No consumers show in burrow-ui, burrow-dashboard or by curling burrow HTTP endpoint:
```
$ docker logs -f burrow
Reading configuration from /etc/burrow
{"level":"info","ts":1561039131.3334801,"msg":"Started Burrow"}
{"level":"info","ts":1561039131.333604,"msg":"configuring","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1561039131.333678,"msg":"configuring","type":"coordinator","name":"storage"}
{"level":"info","ts":1561039131.3337097,"msg":"configuring","type":"module","coordinator":"storage","class":"inmemory","name":"mystorage"}
{"level":"info","ts":1561039131.3338077,"msg":"configuring","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1561039131.3338366,"msg":"configuring","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1561039131.3338754,"msg":"configuring","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1561039131.333965,"msg":"configuring","type":"coordinator","name":"notifier"}
{"level":"info","ts":1561039131.334004,"msg":"configuring","type":"coordinator","name":"cluster"}
{"level":"info","ts":1561039131.33403,"msg":"configuring","type":"module","coordinator":"cluster","class":"kafka","name":"hdp30"}
{"level":"info","ts":1561039131.3341236,"msg":"configuring","type":"coordinator","name":"consumer"}
{"level":"info","ts":1561039131.3341634,"msg":"configuring","type":"module","coordinator":"consumer","class":"kafka_zk","name":"hdp-burrow30"}
{"level":"info","ts":1561039131.3342574,"msg":"starting","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1561039131.3352957,"msg":"Connected to 10.207.180.134:2181","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1561039131.336034,"msg":"Authenticated: id=174272131742600143, timeout=6000","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1561039131.3360665,"msg":"Re-submitting `0` credentials after reconnect","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1561039131.3410175,"msg":"starting","type":"coordinator","name":"storage"}
{"level":"info","ts":1561039131.3410745,"msg":"starting","type":"module","coordinator":"storage","class":"inmemory","name":"mystorage"}
{"level":"info","ts":1561039131.3413737,"msg":"starting","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1561039131.341399,"msg":"starting","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1561039131.3414104,"msg":"starting","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1561039131.3640947,"msg":"started listener","type":"coordinator","name":"httpserver","listener":"0.0.0.0:8000"}
{"level":"info","ts":1561039131.364167,"msg":"starting","type":"coordinator","name":"notifier"}
{"level":"info","ts":1561039131.3641877,"msg":"starting","type":"coordinator","name":"cluster"}
{"level":"info","ts":1561039131.3641932,"msg":"starting","type":"module","coordinator":"cluster","class":"kafka","name":"hdp30"}
{"level":"info","ts":1561039131.402523,"msg":"starting","type":"coordinator","name":"consumer"}
{"level":"info","ts":1561039131.4025538,"msg":"starting","type":"module","coordinator":"consumer","class":"kafka_zk","name":"hdp-burrow30"}
{"level":"info","ts":1561039131.4047937,"msg":"Connected to 10.207.180.134:2181","type":"module","coordinator":"consumer","class":"kafka_zk","name":"hdp-burrow30"}
{"level":"info","ts":1561039131.4054527,"msg":"Authenticated: id=174272131742600144, timeout=6000","type":"module","coordinator":"consumer","class":"kafka_zk","name":"hdp-burrow30"}
{"level":"info","ts":1561039131.4054873,"msg":"Re-submitting `0` credentials after reconnect","type":"module","coordinator":"consumer","class":"kafka_zk","name":"hdp-burrow30"}
{"level":"info","ts":1561039131.4660237,"msg":"starting evaluations","type":"coordinator","name":"notifier"}
{"level":"info","ts":1561039131.6079664,"msg":"cluster or consumer not found","type":"module","coordinator":"evaluator","class":"caching","name":"default","cluster":"hdp30","consumer":"console-consumer-59921","showall":true}
{"level":"info","ts":1561039131.6086192,"msg":"cluster or consumer not found","type":"module","coordinator":"evaluator","class":"caching","name":"default","cluster":"hdp30","consumer":"console-consumer-66028","showall":true}
{"level":"info","ts":1561039131.6089625,"msg":"cluster or consumer not found","type":"module","coordinator":"evaluator","class":"caching","name":"default","cluster":"hdp30","consumer":"console-consumer-17809","showall":true}
{"level":"info","ts":1561039131.6092784,"msg":"cluster or consumer not found","type":"module","coordinator":"evaluator","class":"caching","name":"default","cluster":"hdp30","consumer":"console-consumer-17714","showall":true}
{"level":"info","ts":1561039131.6096077,"msg":"cluster or consumer not found","type":"module","coordinator":"evaluator","class":"caching","name":"default","cluster":"hdp30","consumer":"console-consumer-91917","showall":true}
{"level":"info","ts":1561039131.610201,"msg":"cluster or consumer not found","type":"module","coordinator":"evaluator","class":"caching","name":"default","cluster":"hdp30","consumer":"console-consumer-89795","showall":true}
{"level":"info","ts":1561039131.6121168,"msg":"cluster or consumer not found","type":"module","coordinator":"evaluator","class":"caching","name":"default","cluster":"hdp30","consumer":"console-consumer-68185","showall":true}
{"level":"info","ts":1561039131.6126003,"msg":"cluster or consumer not found","type":"module","coordinator":"evaluator","class":"caching","name":"default","cluster":"hdp30","consumer":"console-consumer-94287","showall":true}
```
So, question is: is it because offsets are not being written in __consumer_offset ?
Thanks

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the kafka_zk consumer module and its handling of the configured Zookeeper paths, then compare the reported configuration with where Kafka stores consumer offsets. Reproduce the setup using the supplied burrow.toml and logs; done means the configured consumers appear through the Burrow HTTP endpoint and UI.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.