Question: Total vs Partition Lag
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
I have a question on proper monitoring of Kafka group/topic lag using the endpoint referenced below, my objective is to figure out when a consumer isn't consuming outside of it's given threashold. Some apps that may be 1MM messages, once a week.. Other apps that should never be more than 2k messages deep over 5m.
I looked at:
v3/kafka/(cluster)/consumer/(group)/lag
In there I see two interesting elements:
```status.totallag ```
and
```
status.partition[0..n].start.lag
status.partition[0..n].current_lag
```
Ideally, I would like to use status.totallag, however, when I tally up, that value never matches either start.lag or current_lag. Especially when there is a great deal of lag for the consumer group.
In addition we are using a burrow-->prometheus exporter to generate a scrape with the above referenced route.
Thank you for your time in answering this question.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the v3/kafka/(cluster)/consumer/(group)/lag endpoint and compare status.totallag with each partition's start.lag and current_lag, including the Burrow-to-Prometheus exporter output. Done means documenting why these values differ and which metric best supports the stated consumer-threshold monitoring cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka, prometheus
- Domain
- api, observability, stream-processing
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100