linkedin / linkedin/Burrow

Confluent Cloud Setup

Ouverte
#653 6 commentaires 7 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Langage dominant
Go
Étoiles
4k
Forks
818
Merge moyen
1 h 14 min
PR mergées (30 j)
1

Description

Hi guys,

I've been struggling with the burrow setup for quite some time for kafka cluster on confluent cloud. Burrow has been really valuable piece of software for the inhouse deployed kafka clusters but since my team moved to the confluent cloud I've been struggling to set it up correctly.

Here's the config:
```
[zookeeper]
servers=[ "172.17.0.4:2181" ]
timeout=6
root-path="/burrow"

[logging]
level="debug"

[cluster.stg]
class-name="kafka"
servers=[ "*****.us-east1.gcp.confluent.cloud:9092" ]
client-profile="myclient"
topic-refresh=60
offset-refresh=30

[consumer.stg]
class-name="kafka"
cluster="stg"
client-profile="myclient"
servers=[ "*****.us-east1.gcp.confluent.cloud:9092" ]
group-allowlist=""

[client-profile.myclient]
kafka-version="0.11.0"
client-id="burrow-myclient"
sasl="mysaslprofile"

[sasl.mysaslprofile]
username="*********"
password="*********"
handshake-first=false

[httpserver.default]
address=":8000"
```

I was trying all possible config combinations and the one above is the one that got me to pass the SASL auth but it's still failing as client metadata gets EOF error. Please see below:
```
Reading configuration from /etc/burrow
{"level":"info","ts":1597737468.3371112,"msg":"Started Burrow"}
{"level":"info","ts":1597737468.3375235,"msg":"configuring","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.3385446,"msg":"configuring","type":"coordinator","name":"storage"}
{"level":"info","ts":1597737468.3386433,"msg":"configuring","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1597737468.3388054,"msg":"configuring","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1597737468.3388784,"msg":"configuring","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1597737468.3389807,"msg":"configuring","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1597737468.3399022,"msg":"configuring","type":"coordinator","name":"notifier"}
{"level":"info","ts":1597737468.3400364,"msg":"configuring","type":"coordinator","name":"cluster"}
{"level":"info","ts":1597737468.3400853,"msg":"configuring","type":"module","coordinator":"cluster","class":"kafka","name":"oc_prod"}
{"level":"info","ts":1597737468.3452342,"msg":"configuring","type":"coordinator","name":"consumer"}
{"level":"info","ts":1597737468.3453486,"msg":"configuring","type":"module","coordinator":"consumer","class":"kafka","name":"prod"}
{"level":"info","ts":1597737468.3459487,"msg":"starting","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.3561957,"msg":"Connected to 172.17.0.4:2181","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.4135559,"msg":"authenticated: id=72138833186127922, timeout=6000","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.4136565,"msg":"re-submitting `0` credentials after reconnect","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737468.4246602,"msg":"starting","type":"coordinator","name":"storage"}
{"level":"info","ts":1597737468.42482,"msg":"starting","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1597737468.4251058,"msg":"starting","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1597737468.4251423,"msg":"starting","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1597737468.425219,"msg":"starting","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1597737468.4268248,"msg":"started listener","type":"coordinator","name":"httpserver","listener":"[::]:8000"}
{"level":"info","ts":1597737468.427205,"msg":"starting","type":"coordinator","name":"notifier"}
{"level":"info","ts":1597737468.4276655,"msg":"starting","type":"coordinator","name":"cluster"}
{"level":"info","ts":1597737468.4277368,"msg":"starting","type":"module","coordinator":"cluster","class":"kafka","name":"oc_prod"}
{"level":"debug","ts":1597737468.4280841,"msg":"Initializing new client","name":"sarama"}
{"level":"debug","ts":1597737468.4307697,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"info","ts":1597737468.5562336,"msg":"starting evaluations","type":"coordinator","name":"notifier"}
{"level":"debug","ts":1597737468.7360644,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737468.7361805,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737468.738494,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737468.7386537,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737468.7387395,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737468.7387767,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737468.7388234,"msg":"client/metadata retrying after 250ms... (3 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1597737468.989536,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.2316642,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737469.231808,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737469.2322161,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737469.232594,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.2328062,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737469.2328613,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737469.2329073,"msg":"client/metadata retrying after 250ms... (2 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1597737469.483665,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.7172434,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737469.7200181,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737469.7209947,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737469.7211635,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737469.721262,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737469.7216485,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737469.7218363,"msg":"client/metadata retrying after 250ms... (1 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1597737469.972495,"msg":"client/metadata fetching metadata for all topics from broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737470.2147756,"msg":"SASL authentication successful with broker ******.us-east1.gcp.confluent.cloud:9092:4 - [21 3 3 0]","name":"sarama"}
{"level":"debug","ts":1597737470.2148404,"msg":"Connected to broker at ******.us-east1.gcp.confluent.cloud:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1597737470.2154193,"msg":"client/metadata got error from broker -1 while fetching metadata: unexpected EOF","name":"sarama"}
{"level":"debug","ts":1597737470.215738,"msg":"Closed connection to broker ******.us-east1.gcp.confluent.cloud:9092","name":"sarama"}
{"level":"debug","ts":1597737470.2158542,"msg":"client/metadata no available broker to send metadata request to","name":"sarama"}
{"level":"debug","ts":1597737470.2159607,"msg":"client/brokers resurrecting 1 dead seed brokers","name":"sarama"}
{"level":"debug","ts":1597737470.216271,"msg":"Closing Client","name":"sarama"}
{"level":"error","ts":1597737470.2164073,"msg":"failed to start client","type":"module","coordinator":"cluster","class":"kafka","name":"oc_prod","error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
{"level":"info","ts":1597737470.2164896,"msg":"stopping","type":"coordinator","name":"notifier"}
{"level":"info","ts":1597737470.2165515,"msg":"shutdown","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1597737470.2169995,"msg":"stopping","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1597737470.217062,"msg":"stopping","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1597737470.217476,"msg":"stopping","type":"coordinator","name":"storage"}
{"level":"info","ts":1597737470.2175245,"msg":"stopping","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1597737470.2175949,"msg":"stopping","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737470.22605,"msg":"recv loop terminated: err=EOF","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1597737470.2261844,"msg":"send loop terminated: err=","type":"coordinator","name":"zookeeper"}
Burrow failed at August 18, 2020 at 7:57am (UTC)
```

I was wondering if any of you had any experience in setting up burrow against managed kafka cluster in the confluent cloud. It seems to me this is something with either confluent cloud design or my cluster config but it's really strange that this example is not documented anywhere. I guess managed kafka service will be getting more adoption so documenting this use case will be valuable for the wider community.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Examinez la configuration Confluent Cloud présentée dans l’issue avec la documentation du client Kafka et des profils client de Burrow. Commencez par les logs EOF de métadonnées de Sarama et déterminez si cette configuration est prise en charge ou si elle nécessite des indications de configuration documentées. Le travail est terminé lorsqu’une configuration Confluent Cloud claire et reproductible ou une explication explicite de la limitation est disponible.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
go
Domaine
distributed-systems, documentation
Type d'issue
Documentation
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
30/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.