kafka: client has run out of available brokers to talk to (Is your cluster reachable?

**Environment:**

```
OS: RHEL 7.3
Docker 1.13.1-cs4
Burrow: latest
Kafka: Confluent-3.3.0
Authentication: SASL_PLAINTEXT
```

**Problem:**

Container-built Burrow cannot connect to Kafka nodes

```
{"level":"error","ts":1522848692.2069006,"msg":"failed to start client","type":"module","coordinator":"cluster","class":"kafka","name":"dp_cnj_kafka_prod","error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
```

**Steps to reproduce:**

Clone repository

```git clone https://github.com/linkedin/Burrow```

Reconfigure configuration file: ```burrow.toml```

```
[general]
pidfile="burrow.pid"
stdout-logfile="burrow.out"
access-control-allow-origin="*"

[logging]
filename="/var/log/burrow.log"
level="debug"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true

[zookeeper]
servers=["server-cnj-prod-z1:2181","server-cnj-prod-z2:2181","server-cnj-prod-z3:2181"]
timeout=6
zookeeper-offsets=true

[cluster.dp_cnj_kafka_prod]
client-profile="dp_cnj_kafka_prod"
class-name="kafka"
servers=["server-cnj-prod-k1:9092","server-cnj-prod-k2:9092","server-cnj-prod-k3:9092"]
topic-refresh=120
offset-refresh=30
offset-topic="__consumer_offsets"

[storage.default]
class-name="inmemory"
workers=20
intervals=15
expire-group=604800
min-distance=1

[client-profile.dp_cnj_kafka_prod]
kafka-version="0.11.0"
client-id="burrow-client"
sasl="sasl_dp_cnj_kafka"

[sasl.sasl_dp_cnj_kafka]
username="admin"
password="XXXXXXXXXXXXX"
handshake-first=false

[consumer.consumer_dp_cnj_kafka]
class-name="kafka"
cluster="dp_cnj_kafka_prod"
client-profile="dp_cnj_kafka_prod"
servers=["server-cnj-prod-k1:9092","server-cnj-prod-k2:9092","server-cnj-prod-k3:9092"]
start_latest=false
offset_topic="__consumer_offsets"
group-whitelist=".*"
group-blacklist="^(console-consumer-|python-kafka-consumer-).*$"

[consumer.consumer_dp_cnj_zk]
class-name="kafka_zk"
cluster="dp_cnj_kafka_prod"
servers=["server-cnj-prod-z1:2181","server-cnj-prod-z2:2181","server-cnj-prod-z3:2181"]
zookeeper-timeout=30
group-blacklist="^(console-consumer-|python-kafka-consumer-).*$"

[httpserver.default]
address=":8000"
```

################################################################

Build container:

```
docker build -t akamalov/burrow:1.0 .
```
Deploy container with a volume map where configuration directory is mapped to a container, as well as log directory:

```
docker run -d -v /opt/burrow:/etc/burrow -v /opt/burrow/logs:/var/log/ -p 8000:8000 akamalov/burrow:1.0
```
################################################################

**burrow.log** output:

```
{"level":"info","ts":1522848690.73516,"msg":"Started Burrow"}
{"level":"info","ts":1522848690.7353127,"msg":"configuring","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1522848690.7359643,"msg":"configuring","type":"coordinator","name":"storage"}
{"level":"info","ts":1522848690.7359943,"msg":"configuring","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1522848690.7360544,"msg":"configuring","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1522848690.7360692,"msg":"configuring","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1522848690.7360852,"msg":"configuring","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1522848690.7361557,"msg":"configuring","type":"coordinator","name":"notifier"}
{"level":"info","ts":1522848690.7361748,"msg":"configuring","type":"coordinator","name":"cluster"}
{"level":"info","ts":1522848690.7361956,"msg":"configuring","type":"module","coordinator":"cluster","class":"kafka","name":"dp_cnj_kafka_prod"}
{"level":"info","ts":1522848690.7367117,"msg":"configuring","type":"coordinator","name":"consumer"}
{"level":"info","ts":1522848690.7367504,"msg":"configuring","type":"module","coordinator":"consumer","class":"kafka","name":"consumer_dp_cnj_kafka"}
{"level":"info","ts":1522848690.7372715,"msg":"configuring","type":"module","coordinator":"consumer","class":"kafka_zk","name":"consumer_dp_cnj_zk"}
{"level":"info","ts":1522848690.7377377,"msg":"starting","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1522848690.7678626,"msg":"Connected to 192.168.2.27:2181","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1522848690.7947993,"msg":"Authenticated: id=171768459403460873, timeout=6000","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1522848690.794854,"msg":"Re-submitting `0` credentials after reconnect","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1522848690.8221726,"msg":"starting","type":"coordinator","name":"storage"}
{"level":"info","ts":1522848690.8222377,"msg":"starting","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1522848690.822296,"msg":"starting","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1522848690.8223126,"msg":"starting","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1522848690.8223212,"msg":"starting","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1522848690.8226066,"msg":"started listener","type":"coordinator","name":"httpserver","listener":"[::]:8000"}
{"level":"info","ts":1522848690.8226492,"msg":"starting","type":"coordinator","name":"notifier"}
{"level":"info","ts":1522848690.8226643,"msg":"starting","type":"coordinator","name":"cluster"}
{"level":"info","ts":1522848690.8227122,"msg":"starting","type":"module","coordinator":"cluster","class":"kafka","name":"dp_cnj_kafka_prod"}
{"level":"info","ts":1522848690.9740508,"msg":"starting evaluations","type":"coordinator","name":"notifier"}
{"level":"error","ts":1522848692.2069006,"msg":"failed to start client","type":"module","coordinator":"cluster","class":"kafka","name":"dp_cnj_kafka_prod","error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
{"level":"info","ts":1522848692.206969,"msg":"stopping","type":"coordinator","name":"notifier"}
{"level":"info","ts":1522848692.2069814,"msg":"shutdown","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1522848692.2070007,"msg":"stopping","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1522848692.207008,"msg":"stopping","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1522848692.207018,"msg":"stopping","type":"coordinator","name":"storage"}
{"level":"info","ts":1522848692.207027,"msg":"stopping","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1522848692.2071266,"msg":"stopping","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1522848692.2337039,"msg":"Recv loop terminated: err=EOF","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1522848692.2337453,"msg":"Send loop terminated: err=","type":"coordinator","name":"zookeeper"}
```

It can't seem to be able to reach kafka nodes.

- All iptables rules are flushed (source and destination)
- Nodes are completely routable:

```
$ telnet server-cnj-prod-k1 9092
Trying 192.168.2.25...
Connected to server-cnj-prod-k1.
Escape character is '^]'.
^C
Connection closed by foreign host.
```

Any ideas ? Your help is very much appreciated. Thank you.

################################################################

Start by reproducing the reported Docker build and run with the supplied burrow.toml, then compare the Kafka client and SASL settings with the container logs and telnet result. Done should identify whether the failure is caused by configuration or container connectivity and document a reproducible fix.

Written by the indexing model from the issue text.