linkedin / linkedin/Burrow

Burrow fail to retrieve some consumer group data when zk timeout occurs

Open
#520 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

When burrow connect zk timeout and reconnect,it fails to retrieve some group offset data。
My burrow commit version is 12e681a3a8a61f84f17677996dc3e6a2b79fac41 "Tue Jun 26 14:51:53 2018 +0530"
The related log in burrow.log:

{"level":"info","ts":1555956891.5007114,"msg":"Recv loop terminated: err=read tcp 10.100.9.187:22204->10.224.1.59:2181: i/o timeout","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}
{"level":"info","ts":1555956891.5007553,"msg":"Send loop terminated: err=","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}
{"level":"warn","ts":1555956891.5008717,"msg":"failed to read offset","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk","group":"cj_room.collection.logic_prod_maweijie.liveroom_buz_inout.roomcollect","topic":"liveroom_buz_inout","partition":1,"error":"zk: connection closed"}
{"level":"warn","ts":1555956891.5008936,"msg":"failed to read offset","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk","group":"cj_room.link.social_cj_hall.homepage.logic_prod_yuwenxi.liveroom_buz_inout","topic":"liveroom_buz_inout","partition":1,"error":"zk: connection closed"}
{"level":"warn","ts":1555956891.500902,"msg":"failed to read offset","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk","group":"cj_room.link.social_cj_hall.homepage.logic_prod_yuwenxi.liveroom_buz_link","topic":"liveroom_buz_link","partition":9,"error":"zk: connection closed"}
{"level":"info","ts":1555956891.5021763,"msg":"Connected to 10.224.1.69:2181","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}
{"level":"info","ts":1555956891.5036764,"msg":"Authenticated: id=317466785781646635, timeout=30000","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}
{"level":"info","ts":1555956891.5036852,"msg":"Re-submitting `0` credentials after reconnect","type":"module","coordinator":"consumer","class":"kafka_zk","name":"tx-cj_kafka01_consumer_offset_in_zk"}

Then the group cj_room.collection.logic_prod_maweijie.liveroom_buz_inout.roomcollect on "liveroom_buz_inoucj_room.link.social_cj_hall.homepage.logic_prod_yuwenxi.liveroom_buz_link-1 、cj_room.link.social_cj_hall.homepage.logic_prod_yuwenxi.liveroom_buz_inout on "liveroom_buz_inout-1 will fail to get offset unless restart burrow

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the consumer coordinator failure using the reported ZooKeeper timeout and reconnect logs, then trace the kafka_zk offset retrieval path. The issue is fixed when the affected consumer groups recover their offsets after reconnection without restarting Burrow.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.