Burrow fails to start with Kafka Version 2.8.x
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
Hi team,
I am using Burrow 1.3.6 which works fine with Kafka Version 2.7 but when I upgrade Kafka to version 2.8.1, burrow fails to start with the following error:
```
{"level":"debug","ts":1642027415.5584846,"msg":"Connected to broker at uem-kafka:9092 (unregistered)","name":"sarama"}
{"level":"debug","ts":1642027415.5632129,"msg":"client/brokers registered new broker #0 at uem-kafka:9092","name":"sarama"}
{"level":"debug","ts":1642027415.5634372,"msg":"Successfully initialized new client","name":"sarama"}
{"level":"debug","ts":1642027415.5635583,"msg":"client/metadata fetching metadata for [__consumer_offsets] from broker uem-kafka:9092","name":"sarama"}
{"level":"debug","ts":1642027415.5732005,"msg":"client/metadata found some partitions to be leaderless","name":"sarama"}
{"level":"debug","ts":1642027415.5733514,"msg":"client/metadata retrying after 250ms... (3 attempts remaining)","name":"sarama"}
{"level":"info","ts":1642027415.6499023,"msg":"starting evaluations","type":"coordinator","name":"notifier"}
{"level":"debug","ts":1642027415.8234534,"msg":"client/metadata fetching metadata for [__consumer_offsets] from broker uem-kafka:9092","name":"sarama"}
{"level":"debug","ts":1642027415.8260822,"msg":"client/metadata found some partitions to be leaderless","name":"sarama"}
{"level":"debug","ts":1642027415.8263423,"msg":"client/metadata retrying after 250ms... (2 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1642027416.076764,"msg":"client/metadata fetching metadata for [__consumer_offsets] from broker uem-kafka:9092","name":"sarama"}
{"level":"debug","ts":1642027416.0788453,"msg":"client/metadata found some partitions to be leaderless","name":"sarama"}
{"level":"debug","ts":1642027416.0788953,"msg":"client/metadata retrying after 250ms... (1 attempts remaining)","name":"sarama"}
{"level":"debug","ts":1642027416.3291106,"msg":"client/metadata fetching metadata for [__consumer_offsets] from broker uem-kafka:9092","name":"sarama"}
{"level":"debug","ts":1642027416.3314283,"msg":"client/metadata found some partitions to be leaderless","name":"sarama"}
Burrow failed at January 12, 2022 at 10:43pm (UTC)
{"level":"error","ts":1642027416.3314698,"msg":"failed to get partition count","type":"module","coordinator":"consumer","class":"kafka","name":"local-kafka","topic":"__consumer_offsets","error":"kafka server: Request was for a topic or partition that does not exist on this broker."}
{"level":"debug","ts":1642027416.3315206,"msg":"Closing Client","name":"sarama"}
{"level":"error","ts":1642027416.3315384,"msg":"failed to start consumer","type":"module","coordinator":"consumer","class":"kafka","name":"local-kafka","error":"kafka server: Request was for a topic or partition that does not exist on this broker."}
{"level":"debug","ts":1642027416.3315632,"msg":"Close() called on already closed client","name":"sarama"}
```
Sample of docker-compose.yml for Kafka and Zookeeper service:
```
version: '3.7'
services:
uem-kafka:
image: bitnami/kafka:2.8.1
ports:
- '9092:9092'
user: "1001"
environment:
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_BROKER_ID=0
- KAFKA_CFG_NUM_PARTITIONS=3
- KAFKA_CFG_ZOOKEEPER_CONNECT=uem-zookeeper:2181
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
- KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INSIDE
- "KAFKA_CFG_LISTENERS=INSIDE://uem-kafka:9094,OUTSIDE://uem-kafka:9092"
- "KAFKA_CFG_ADVERTISED_LISTENERS=INSIDE://uem-kafka:9094,OUTSIDE://uem-kafka:9092"
- KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=false
depends_on:
- uem-zookeeper
uem-zookeeper:
image: bitnami/zookeeper:3.6.2
container_name: uem-zookeeper
ports:
- '2181:2181'
environment:
- ZOO_SERVER_ID=0
- ZOO_HEAP_SIZE=100
- ALLOW_ANONYMOUS_LOGIN=yes
- ZOO_4LW_COMMANDS_WHITELIST="*"
- ZOO_SERVERS='0.0.0.0:2888:3888'
```
Is burrow with Kafka Version 2.8.x supported? Any suggestions on how to troubleshoot this further?
Please let me know if there is any other log or configuration required.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing Burrow 1.3.6's Kafka consumer startup path and the supplied docker-compose.yml, then reproduce against Kafka 2.8.1 using the reported Sarama logs. Check whether the failure is a compatibility problem or a configuration/state issue. Done means identifying the cause and documenting a confirmed fix or supported-version limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, go, kafka
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100