linkedin / linkedin/Burrow

consumer offset not updated again after "failed to read offset" error

Open
#384 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Once a week or so, we notice that some consumers are falling way behind. We check the consumers in Kafka Manager and it reports that eveything is ok. When we look at the burrow logs around the time the ramp up started, we see a message similar to this one:

> {"level":"warn","ts":1523647869.7964308,"msg":"failed to read offset","type":"module","coordinator":"consumer","class":"kafka_zk","name":"local_zk","group":"logstash_telegraph","topic":"telegraf_json","partition":0,"error":"zk: zookeeper is closing"}

It never seems to fix itself. Our only recourse is to reboot burrow, after which it reports everything is OK.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Burrow's consumer offset handling around the logged "failed to read offset" error, especially the kafka_zk consumer coordinator and the affected consumer group, topic, and partition. Reproduce or inspect the behavior after ZooKeeper reports that it is closing, and consider the issue done when offset updates resume without requiring a Burrow reboot.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
observability, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.