Ingestion delay metrics should output a tag for the topic name
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
With multi-topics ingestion from Pinot now, it would be helpful the delay metrics emit the topic name as well. At the moment, if we want to find which topic has the issue, we have to do partition % 1000 and get the kafka topic name from the config. This is especially an issue with the new kafka offset reset feature since in this case Pinot automatically adds topics to the config. If we want to just get the backfill topics ingestion or only exclude them, there's no way to do it.
Contributor guide
Research direction
The issue does not name a file or test; start by locating the ingestion delay metric emission and the code that associates records with Kafka topics. Done means the delay metrics include the topic name as a tag, allowing backfill topics or individual topics to be identified and filtered without deriving names from partition numbers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100