Consumer status is unpredictable when multiple topics are consumed
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
https://github.com/linkedin/Burrow/blob/4a05b203ce40d82def3af986eeca5350570e6b96/core/internal/evaluator/caching.go#L226-L252
This piece of code loops over a map of topics, and if the last topic's last partition is reporting ok, the consumer status will be ok.
Given that the map iteration in go is randomized, the consumer status is unpredictable.
The following are the real world effect from this:
1. The metric from burrow of a consumer when scraping at 2m interval:
2. The metric from burrow-exporter which requests burrow at 30s interval, and then being scrapped at 2m interval:
The more frequently we query (as burrow uses 30s cache expiration by default), the more likely to see non-OK consumer status.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in core/internal/evaluator/caching.go around lines 226-252, where consumer status is calculated while iterating over topic partitions. Trace how status values are combined and inspect nearby evaluator behavior before deciding the expected aggregate result. Done means repeated evaluations produce a deterministic consumer status that accounts for all consumed topics and partitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100