linkedin / linkedin/Burrow

Configure minimum lag at 1 (and not 0) ?

Open
#570 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

I try to use Burrow to monitor a bunch of services build with robinhood [*faust* python library](https://faust.readthedocs.io/).

The point is that *faust* has a different understanding of kafka consumer's offset. For *faust* offset is the last event offset, not the "next expected event" offset. see https://github.com/robinhood/faust/issues/73

So a *faust* based services always have a minimal lag of 1 (and not 0). And so burrow set an error status when there is no commit for some times...

In my point of view the issue comes from *faust* interpretation of these offset.
so => https://github.com/robinhood/faust/issues/73

But, anyway, I post this for two reasons :
- for documentation (I couldn't found any reference of this issue on burrow github)
- I wounder if there is a -not-so-dirty- way to configure burrow to handle this minimal lag at 1 ?

btw, thanks for you work on burrow !

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading Burrow's configuration and consumer-lag status handling, then compare its offset assumptions with the Faust offset behavior described in the issue and linked discussion. Done means determining whether a minimum lag of 1 can be configured, documenting the supported approach, or clearly documenting that the behavior is not configurable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.