consumer offset not updated again after "failed to read offset" error
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
Once a week or so, we notice that some consumers are falling way behind. We check the consumers in Kafka Manager and it reports that eveything is ok. When we look at the burrow logs around the time the ramp up started, we see a message similar to this one:
> {"level":"warn","ts":1523647869.7964308,"msg":"failed to read offset","type":"module","coordinator":"consumer","class":"kafka_zk","name":"local_zk","group":"logstash_telegraph","topic":"telegraf_json","partition":0,"error":"zk: zookeeper is closing"}
It never seems to fix itself. Our only recourse is to reboot burrow, after which it reports everything is OK.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Burrow's consumer offset handling around the logged "failed to read offset" error, especially the kafka_zk consumer coordinator and the affected consumer group, topic, and partition. Reproduce or inspect the behavior after ZooKeeper reports that it is closing, and consider the issue done when offset updates resume without requiring a Burrow reboot.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- observability, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100