Improve/Add completeness and freshness metrics for Pinot
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
This issue tracks the need to expose better metrics in Pinot to expose completeness and freshness in the query path. Today, a lot of individual metrics are available for tracking, but its typically hard to say which queries were actually affected when issues do happen. Allowing tracking/metadata to be sent in query response helps with this. This issue tracks efforts being made for this.
NOTE: the tracking is on a best effort basis - we would like to track the most common possibilities (without tying ourselves to specific individual scenarios) without compromising the execution time of the query and continuing to keep the design/implementation as simple/intuitive as possible.
Details:
======
Completeness:
-------------------
Currently, we set a "partial-response" metadata flag when a broker is unable to get a connection to the server or response from a server within configured timeout. We would like to expand this to cover:
(a) cases where servers don't have the segments the broker is requesting.
(b) broker's routing table is incomplete - for ex, when all the replicas of a given segment are ofline or in error state.
Freshness:
---------------
In the current scope, we limit the freshness of the data to realtime tables. We would like to measure how "recent" the data available for querying in realtime is.
More details for each of these will follow.
Contributor guide
Research direction
Start by tracing the existing "partial-response" metadata through Pinot's broker query path and reviewing the individual metrics already exposed. Determine how missing segments, incomplete routing, and realtime data age should be represented without compromising query execution time; the issue does not yet identify specific files, tests, or final acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100