apache / apache/pinot

Ingestion delay metrics should output a tag for the topic name

Open
#18,099 0 comments 0 reactions 0 assignees View on GitHub
feature ingestion metrics observability real-time
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

With multi-topics ingestion from Pinot now, it would be helpful the delay metrics emit the topic name as well. At the moment, if we want to find which topic has the issue, we have to do partition % 1000 and get the kafka topic name from the config. This is especially an issue with the new kafka offset reset feature since in this case Pinot automatically adds topics to the config. If we want to just get the backfill topics ingestion or only exclude them, there's no way to do it.

Contributor guide

Open the contributing guide

Research direction

The issue does not name a file or test; start by locating the ingestion delay metric emission and the code that associates records with Kafka topics. Done means the delay metrics include the topic name as a tag, allowing backfill topics or individual topics to be identified and filtered without deriving names from partition numbers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
backend, observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.