Burrow fail to retrieve some consumer group data when zk timeout occurs
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
When burrow connect zk timeout and reconnect,it fails to retrieve some group offset data。
My burrow commit version is 12e681a3a8a61f84f17677996dc3e6a2b79fac41 "Tue Jun 26 14:51:53 2018 +0530"
The related log in burrow.log:
{"level":"info","ts":1555956891.5007114,"msg":"Recv loop terminated: err=read tcp 10.100.9.187:22204->10.224.1.59:2181: i/o timeout","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}
{"level":"info","ts":1555956891.5007553,"msg":"Send loop terminated: err=","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}
{"level":"warn","ts":1555956891.5008717,"msg":"failed to read offset","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk","group":"cj_room.collection.logic_prod_maweijie.liveroom_buz_inout.roomcollect","topic":"liveroom_buz_inout","partition":1,"error":"zk: connection closed"}
{"level":"warn","ts":1555956891.5008936,"msg":"failed to read offset","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk","group":"cj_room.link.social_cj_hall.homepage.logic_prod_yuwenxi.liveroom_buz_inout","topic":"liveroom_buz_inout","partition":1,"error":"zk: connection closed"}
{"level":"warn","ts":1555956891.500902,"msg":"failed to read offset","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk","group":"cj_room.link.social_cj_hall.homepage.logic_prod_yuwenxi.liveroom_buz_link","topic":"liveroom_buz_link","partition":9,"error":"zk: connection closed"}
{"level":"info","ts":1555956891.5021763,"msg":"Connected to 10.224.1.69:2181","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}
{"level":"info","ts":1555956891.5036764,"msg":"Authenticated: id=317466785781646635, timeout=30000","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}
{"level":"info","ts":1555956891.5036852,"msg":"Re-submitting `0` credentials after reconnect","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}
Then the group cj_room.collection.logic_prod_maweijie.liveroom_buz_inout.roomcollect on "liveroom_buz_inoucj_room.link.social_cj_hall.homepage.logic_prod_yuwenxi.liveroom_buz_link-1 、cj_room.link.social_cj_hall.homepage.logic_prod_yuwenxi.liveroom_buz_inout on "liveroom_buz_inout-1 will fail to get offset unless restart burrow
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the consumer coordinator failure using the reported ZooKeeper timeout and reconnect logs, then trace the kafka_zk offset retrieval path. The issue is fixed when the affected consumer groups recover their offsets after reconnection without restarting Burrow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100