linkedin / linkedin/Burrow

Improve lag evaluation rules

Open
#303 21 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

I'm opening this issue to start conversation around possible improvement for the lag detection rules.

Currently Burrow does a great job of detecting lag when a consumer commits at a defined interval. This allows burrow to have a defined window and all the rules work pretty well. If a consumer writes its offsets every second, burrow will always have a window of 10 seconds. If a consumer only writes its offsets when it consumes data, burrow starts to create pretty random windows which can lead to unpredictable results

### Detecting a Stopped consumer in a reasonable time
If there is a low throughput topic and a consumer has built a window (time between first commit and last) of 2 hours, it will take at least that long for a stop to be detected. Current lag will keep increasing, but no commits in the window are changing so burrow never detects a stop.

Would it make sense to have a rule that applies logic against current lag to help improve this case. Something like `if the last X checks to current lag is increasing or the same (non zero) consider the consumer stopped`

### False positive of a stopped consumer
If there is a low throughput topic, and the consumer on that topic only writes offsets when it consumes, burrow can give a false positive of lag. Let's say that a topic only gets data at night. The last offset it consumed gives the group a window (last commit time minus first) of 20 minutes. The consumer than goes for hours without getting anydata. Burrow doesn't report the group in error because current lag is zero. Based on timing of events the consumer could be reported as lagging when the first message comes in. Burrow updates its offset from the broker before the consumer commits. Rule 4 will mark the consumer as stopped since current lag is no longer zero. The lag immediately clears after the first commit comes in

### Match minDistance to offsetRefresh
Would it make sense to remove the configuration for `minDistance` and have it always match the `offsetRefresh` interval. If consumers are writing faster than the refresh interval, they will always look like they are making progress at some point in the window, and lag might not ever show. This is best shown with an example

With this example 10 messages a second are being written and the consumer is consuming at a rate of 1 record per second, writing offsets every 2 seconds. Burrow is refreshing offsets every 10 secoonds

Time | T0 | T5 | T10 | T12 | T14 | T20
-- | -- | -- | -- | -- | -- | --
Head | 0 | 50 | 100 | 120 | 140 | 200
Committed offset | 0 | 4 | 10 | 12 | 14 | 20
Burrow head | 0 | 0 | 100 | 100 | 100 | 200
Burrow lag | 0 | 0 | 90 | 88 | 86 | 180
Actual lag | 0 | 46 | 90 | 108 | 126 | 180

Head = latest offset on the partition log
Committed Offset = The most recent commit the consumer has written
Burrow head = The offset that burrow thinks is the head of the partition
Burrow lag = The lag that burrow calculated
Actual lag = the amount that the consumer is actually lagging.

With this example you can see it will look like the consumer is always making progress in the window, even though it is lagging.

I think an improvement might be to make sure that `minDistance` is >= to the offset refresh interval. This way the window can only advance once per offset refresh.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files, tests, or entry points are named; start by tracing Burrow’s lag-rule evaluation and the minDistance/offsetRefresh configuration. First settle the proposed stopped-consumer, false-positive, and window-advance semantics with maintainers, then define tests that demonstrate the agreed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
observability-sre, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.