linkedin / linkedin/Burrow

Confluent Cloud Setup

Open
#653 6 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Hi guys,

I've been struggling with the burrow setup for quite some time for kafka cluster on confluent cloud. Burrow has been really valuable piece of software for the inhouse deployed kafka clusters but since my team moved to the confluent cloud I've been struggling to set it up correctly.

Here's the config:
```
[zookeeper]
servers=[ "172.17.0.4:2181" ]
timeout=6
root-path="/burrow"

[logging]
level="debug"

[cluster.stg]
class-name="kafka"
servers=[ "*****.us-east1.gcp.confluent.cloud:9092" ]
client-profile="myclient"
topic-refresh=60
offset-refresh=30

[consumer.stg]
class-name="kafka"
cluster="stg"
client-profile="myclient"
servers=[ "*****.us-east1.gcp.confluent.cloud:9092" ]
group-allowlist=""

[client-profile.myclient]
kafka-version="0.11.0"
client-id="burrow-myclient"
sasl="mysaslprofile"

[sasl.mysaslprofile]
username="*********"
password="*********"
handshake-first=false

[httpserver.default]
address=":8000"
```

I was trying all possible config combinations and the one above is the one that got me to pass the SASL auth but it's still failing as client metadata gets EOF error. Please see below:
```
Reading configuration from /etc/burrow
{"level":"info","ts":1597737468.3371112,"msg":"Started Burrow"}
{"level":"info","ts":1597737468.3375235,"msg":"configuring","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.3385446,"msg":"configuring","type":"coordinator","name":"storage"}
{"level":"info","ts":1597737468.3386433,"msg":"configuring","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1597737468.3388054,"msg":"configuring","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1597737468.3388784,"msg":"configuring","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1597737468.3389807,"msg":"configuring","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1597737468.3399022,"msg":"configuring","type":"coordinator","name":"notifier"}
{"level":"info","ts":1597737468.3400364,"msg":"configuring","type":"coordinator","name":"cluster"}
{"level":"info","ts":1597737468.3400853,"msg":"configuring","type":"module","coordinator":"cluster","class":"kafka","name":"oc_prod"}
{"level":"info","ts":1597737468.3452342,"msg":"configuring","type":"coordinator","name":"consumer"}
{"level":"info","ts":1597737468.3453486,"msg":"configuring","type":"module","coordinator":"consumer","class":"kafka","name":"prod"}
{"level":"info","ts":1597737468.3459487,"msg":"starting","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.3561957,"msg":"Connected to 172.17.0.4:2181","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.4135559,"msg":"authenticated: id=72138833186127922, timeout=6000","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.4136565,"msg":"re-submitting `0` credentials after reconnect","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.4246602,"msg":"starting","type":"coordinator","name":"storage"}
{"level":"info","ts":1597737468.42482,"msg":"starting","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1597737468.4251058,"msg":"starting","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1597737468.4251423,"msg":"starting","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1597737468.425219,"msg":"starting","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1597737468.4268248,"msg":"started listener","type":"coordinator","name":"httpserver","listener":"[::]:8000"}
{"level":"info","ts":1597737468.427205,"msg":"starting","type":"coordinator","name":"notifier"}
{"level":"info","ts":1597737468.4276655,"msg":"starting","type":"coordinator","name":"cluster"}
{"level":"info","ts":1597737468.4277368,"msg":"starting","type":"module","coordinator":"cluster","class":"kafka","name":"oc_prod"}
{"level":"debug","ts":1597737468.4280841,"msg":"Initializing new client","name":"sarama"}
{"level":"debug","ts":1597737468.4307697,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"info","ts":1597737468.5562336,"msg":"starting evaluations","type":"coordinator","name":"notifier"}
{"level":"debug","ts":1597737468.7360644,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737468.7361805,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737468.738494,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737468.7386537,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737468.7387395,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737468.7387767,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737468.7388234,"msg":"client/metadata retrying after 250ms... (3 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1597737468.989536,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.2316642,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737469.231808,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737469.2322161,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737469.232594,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.2328062,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737469.2328613,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737469.2329073,"msg":"client/metadata retrying after 250ms... (2 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1597737469.483665,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.7172434,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737469.7200181,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737469.7209947,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737469.7211635,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.721262,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737469.7216485,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737469.7218363,"msg":"client/metadata retrying after 250ms... (1 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1597737469.972495,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737470.2147756,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737470.2148404,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737470.2154193,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737470.215738,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737470.2158542,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737470.2159607,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737470.216271,"msg":"Closing Client","name":"sarama"}
{"level":"error","ts":1597737470.2164073,"msg":"failed to start client","type":"module","coordinator":"cluster","class":"kafka","name":"oc_prod","error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
{"level":"info","ts":1597737470.2164896,"msg":"stopping","type":"coordinator","name":"notifier"}
{"level":"info","ts":1597737470.2165515,"msg":"shutdown","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1597737470.2169995,"msg":"stopping","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1597737470.217062,"msg":"stopping","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1597737470.217476,"msg":"stopping","type":"coordinator","name":"storage"}
{"level":"info","ts":1597737470.2175245,"msg":"stopping","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1597737470.2175949,"msg":"stopping","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737470.22605,"msg":"recv loop terminated: err=EOF","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737470.2261844,"msg":"send loop terminated: err=","type":"coordinator","name":"zookeeper"}
Burrow failed at August 18, 2020 at 7:57am (UTC)
```

I was wondering if any of you had any experience in setting up burrow against managed kafka cluster in the confluent cloud. It seems to me this is something with either confluent cloud design or my cluster config but it's really strange that this example is not documented anywhere. I guess managed kafka service will be getting more adoption so documenting this use case will be valuable for the wider community.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the Confluent Cloud configuration shown in the issue alongside Burrow's Kafka client and client-profile documentation. Start from the Sarama metadata EOF logs and determine whether the setup is supported or needs documented configuration guidance. Done means a clear, reproducible Confluent Cloud setup or an explicit explanation of the limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.