linkedin / linkedin/Burrow

Clarification on WARN for lagging partitions/groups

Open
#738 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Hello,

I'm running a POC for Burrow and can't seem to get a WARN status to appear for my consumer group or any partitions. I've extensively read the [Evaluation Rules](https://github.com/linkedin/Burrow/wiki/Consumer-Lag-Evaluation-Rules), and here's my understanding:

By default, Burrow stores 10 offsets per partition, then using this and the consumer commit interval, determines the size of the evaluation window. I also understand that between _every_ committed offset, the lag needs to either stay the same or increase in order for the group/partition to be considered in a warning state.

I've setup a consumer that is purposely slow, and have a producer that is producing considerably faster. Despite this, Burrow always reports the status to be OK. Even with a lag of 200000+

I keep running over things in my head and trying to tweak my application, but nothing seems to work. I'm hoping someone would be willing to help shed some light on the issue, maybe I'm missing something. Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Burrow's Evaluation Rules and reproduce the report using the described slow consumer and faster producer. Determine why a large lag still reports OK, then document the expected WARN behavior or identify the discrepancy that needs fixing.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.