linkedin / linkedin/Burrow

Consumer status WARN when lag is very low

Open
#223 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

According to the consumer lag evaluation rules - https://github.com/linkedin/Burrow/wiki/Consumer-Lag-Evaluation-Rules, whenever lag increases, the status is set to WARN:

  1. If the consumer offsets are increasing over the window, but the lag either stays the same or increases between every pair of offsets, the consumer is in a WARNING state. This means that the consumer is slow, and is falling behind.

Is there a way to set a threshold for lag in addition to this, my issue is that we trigger alerts on WARNs, but lag is very low - around 200, and clears quickly, whereas we might only want to get a WARN when lag is > 1K or so:

{
"error": false,
"message": "consumer group status returned",
"status": {
"cluster": "local",
"group": "CONSUMER_GROUP",
"status": "WARN",
"complete": true,
"partitions": [
{
"topic": "TOPIC",
"partition": 14,
"status": "WARN",
"start": {
"offset": 112878341,
"timestamp": 1495220705275,
"lag": 11
},
"end": {
"offset": 112882841,
"timestamp": 1495221784466,
"lag": 145
}
},
{
"topic": "TOPIC",
"partition": 23,
"status": "WARN",
"start": {
"offset": 112878287,
"timestamp": 1495220705279,
"lag": 51
},
"end": {
"offset": 112882787,
"timestamp": 1495221784464,
"lag": 185
}
}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the Consumer Lag Evaluation Rules wiki page and tracing how the consumer group status response marks partitions and groups as WARN. Define where a configurable lag threshold would apply, then verify that low-lag increases remain below WARN while lag above the threshold still triggers it.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.