Confluent Cloud Setup

Hi guys,

I've been struggling with the burrow setup for quite some time for kafka cluster on confluent cloud. Burrow has been really valuable piece of software for the inhouse deployed kafka clusters but since my team moved to the confluent cloud I've been struggling to set it up correctly.

Here's the config:

[zookeeper]
servers=[ "172.17.0.4:2181" ]
timeout=6
root-path="/burrow"

[logging]
level="debug"

[cluster.stg]
class-name="kafka"
servers=[ "*****.us-east1.gcp.confluent.cloud:9092" ]
client-profile="myclient"
topic-refresh=60
offset-refresh=30

[consumer.stg]
class-name="kafka"
cluster="stg"
client-profile="myclient"
servers=[ "*****.us-east1.gcp.confluent.cloud:9092" ]
group-allowlist=""

[client-profile.myclient]
kafka-version="0.11.0"
client-id="burrow-myclient"
sasl="mysaslprofile"

[sasl.mysaslprofile]
username="*********"
password="*********"
handshake-first=false

[httpserver.default]
address=":8000"

I was trying all possible config combinations and the one above is the one that got me to pass the SASL auth but it's still failing as client metadata gets EOF error. Please see below:

Reading configuration from /etc/burrow
{"level":"info","ts":1597737468.3371112,"msg":"Started Burrow"}
{"level":"info","ts":1597737468.3375235,"msg":"configuring","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.3385446,"msg":"configuring","type":"coordinator","name":"storage"}
{"level":"info","ts":1597737468.3386433,"msg":"configuring","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1597737468.3388054,"msg":"configuring","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1597737468.3388784,"msg":"configuring","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1597737468.3389807,"msg":"configuring","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1597737468.3399022,"msg":"configuring","type":"coordinator","name":"notifier"}
{"level":"info","ts":1597737468.3400364,"msg":"configuring","type":"coordinator","name":"cluster"}
{"level":"info","ts":1597737468.3400853,"msg":"configuring","type":"module","coordinator":"cluster","class":"kafka","name":"oc_prod"}
{"level":"info","ts":1597737468.3452342,"msg":"configuring","type":"coordinator","name":"consumer"}
{"level":"info","ts":1597737468.3453486,"msg":"configuring","type":"module","coordinator":"consumer","class":"kafka","name":"prod"}
{"level":"info","ts":1597737468.3459487,"msg":"starting","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.3561957,"msg":"Connected to 172.17.0.4:2181","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.4135559,"msg":"authenticated: id=72138833186127922, timeout=6000","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.4136565,"msg":"re-submitting `0` credentials after reconnect","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.4246602,"msg":"starting","type":"coordinator","name":"storage"}
{"level":"info","ts":1597737468.42482,"msg":"starting","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1597737468.4251058,"msg":"starting","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1597737468.4251423,"msg":"starting","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1597737468.425219,"msg":"starting","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1597737468.4268248,"msg":"started listener","type":"coordinator","name":"httpserver","listener":"[::]:8000"}
{"level":"info","ts":1597737468.427205,"msg":"starting","type":"coordinator","name":"notifier"}
{"level":"info","ts":1597737468.4276655,"msg":"starting","type":"coordinator","name":"cluster"}
{"level":"info","ts":1597737468.4277368,"msg":"starting","type":"module","coordinator":"cluster","class":"kafka","name":"oc_prod"}
{"level":"debug","ts":1597737468.4280841,"msg":"Initializing new client","name":"sarama"}
{"level":"debug","ts":1597737468.4307697,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"info","ts":1597737468.5562336,"msg":"starting evaluations","type":"coordinator","name":"notifier"}
{"level":"debug","ts":1597737468.7360644,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737468.7361805,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737468.738494,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737468.7386537,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737468.7387395,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737468.7387767,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737468.7388234,"msg":"client/metadata retrying after 250ms... (3 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1597737468.989536,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.2316642,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737469.231808,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737469.2322161,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737469.232594,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.2328062,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737469.2328613,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737469.2329073,"msg":"client/metadata retrying after 250ms... (2 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1597737469.483665,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.7172434,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737469.7200181,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737469.7209947,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737469.7211635,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.721262,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737469.7216485,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737469.7218363,"msg":"client/metadata retrying after 250ms... (1 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1597737469.972495,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737470.2147756,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737470.2148404,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737470.2154193,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737470.215738,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737470.2158542,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737470.2159607,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737470.216271,"msg":"Closing Client","name":"sarama"}
{"level":"error","ts":1597737470.2164073,"msg":"failed to start client","type":"module","coordinator":"cluster","class":"kafka","name":"oc_prod","error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
{"level":"info","ts":1597737470.2164896,"msg":"stopping","type":"coordinator","name":"notifier"}
{"level":"info","ts":1597737470.2165515,"msg":"shutdown","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1597737470.2169995,"msg":"stopping","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1597737470.217062,"msg":"stopping","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1597737470.217476,"msg":"stopping","type":"coordinator","name":"storage"}
{"level":"info","ts":1597737470.2175245,"msg":"stopping","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1597737470.2175949,"msg":"stopping","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737470.22605,"msg":"recv loop terminated: err=EOF","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737470.2261844,"msg":"send loop terminated: err=<nil>","type":"coordinator","name":"zookeeper"}
Burrow failed at August 18, 2020 at 7:57am (UTC)

I was wondering if any of you had any experience in setting up burrow against managed kafka cluster in the confluent cloud. It seems to me this is something with either confluent cloud design or my cluster config but it's really strange that this example is not documented anywhere. I guess managed kafka service will be getting more adoption so documenting this use case will be valuable for the wider community.

Prüfe die im Issue gezeigte Confluent Cloud-Konfiguration zusammen mit der Kafka-Client- und Client-Profil-Dokumentation von Burrow. Gehe von den Sarama-Metadaten-EOF-Logs aus und ermittle, ob das Setup unterstützt wird oder dokumentierte Konfigurationshinweise benötigt. Als erledigt gilt die Aufgabe, wenn ein klares, reproduzierbares Confluent Cloud-Setup oder eine ausdrückliche Erklärung der Einschränkung vorliegt.