linkedin / linkedin/Burrow

improvement: faster bootstrapping with start-latest=true with full picture of known consumers

Open
#459 0 comments 9 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

when burrow starts it can read offsets from the beginning or the end:

https://github.com/linkedin/Burrow/wiki/Consumer-Kafka

```
start-latest | boolean | no | false |
If true, start consuming the offsets topic from the last (latest) offset.
Otherwise, start from the earliest offset.
```

The advantage of start-latest=true is that burrow will report lag/statuses of consumers that are actively committing fairly quickly. But if a consumer is not committing offsets, it may not be known to burrow and a status may not be available. A burrow restart can lose the "now" view of an inactive consumer.

The advantage of start-latest=false is it will read the entire __consumer_offset topic and build a full picture of all consumers the cluster is storing information on. But that can take a long time depending on how many consumer groups you have, the frequency of commits, and the configuration of the compacted topic. For us this can range from a few minutes to over half an hour during which time burrow reported status may be inaccurate.

but a client can ask the broker for consumer groups and current offsets very quickly, because the brokers cache the latest info: I wonder if this can be improved:

https://godoc.org/github.com/Shopify/sarama#Broker.ListGroups
https://godoc.org/github.com/Shopify/sarama#Broker.DescribeGroups

potentially a new burrow bootstrap could be
* start consuming __consumer_offsets from latest. but don't process anything yet.
* fetch list of groups.
* fetch the details of each group that matches blacklist/whitelist to build the now picture
* then start the processing to keep the picture updated.

Providing both the speed of start-latest=true, and a picture of all consumer groups for "now".

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Burrow's start-latest bootstrap behavior and reading the Sarama Broker.ListGroups and Broker.DescribeGroups APIs linked in the issue. Compare their group and offset information with the blacklist/whitelist requirements; done means startup is fast while the initial now view includes matching known consumer groups and ongoing offset processing remains current.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.