Expose lag metrics when consuming from kafka
Open
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Expose lag metrics when consuming from kafka, this could be sum across all partitions of (approx offset at broker - latest offset consumed) which would be very accurate but should be a close enough approximation.
Contributor guide
Research direction
The issue names no files or tests; start by locating Pinot's Kafka consumer and existing metrics entry points. Confirm how partition offsets are available and define done as exposing a usable lag metric based on the difference between broker and consumed offsets, aggregated across partitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- observability, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100