linkedin / linkedin/Burrow

Consumer status is unpredictable when multiple topics are consumed

Open
#796 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

https://github.com/linkedin/Burrow/blob/4a05b203ce40d82def3af986eeca5350570e6b96/core/internal/evaluator/caching.go#L226-L252

This piece of code loops over a map of topics, and if the last topic's last partition is reporting ok, the consumer status will be ok.

Given that the map iteration in go is randomized, the consumer status is unpredictable.

The following are the real world effect from this:

1. The metric from burrow of a consumer when scraping at 2m interval:
image

2. The metric from burrow-exporter which requests burrow at 30s interval, and then being scrapped at 2m interval:
image

The more frequently we query (as burrow uses 30s cache expiration by default), the more likely to see non-OK consumer status.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in core/internal/evaluator/caching.go around lines 226-252, where consumer status is calculated while iterating over topic partitions. Trace how status values are combined and inspect nearby evaluator behavior before deciding the expected aggregate result. Done means repeated evaluations produce a deterministic consumer status that accounts for all consumed topics and partitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
distributed-systems, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.