apache / apache/pinot

Expose segment reload status as JMX Metric

Open
#6,520 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

It would be very helpful for operation purpose to have the ability to get quick overview on the reload status of a table.

All I could find for now is the `GET /segments/{tableName}_OFFLINE/metadata?type=OFFLINE`. As I'm exploring its output, I realize that the reload status is specific to each individual segments. While providing a startree status there would be a good step forward, a global status would greatly improve the operation experience.

### Output from metadata endpoint

```
"myTable_OFFLINE_1602111600000_1602111600000_2670" : {
"segmentName" : "myTable_OFFLINE_1602111600000_1602111600000_2670",
...
"columns" : [ ],
"indexes" : {
"column1" : {
"bloom-filter" : "NO",
"dictionary" : "YES",
"forward-index" : "YES",
"inverted-index" : "NO",
"null-value-vector-reader" : "NO",
"range-index" : "NO"
},
"column2" : {
"bloom-filter" : "NO",
"dictionary" : "YES",
"forward-index" : "YES",
"inverted-index" : "NO",
"null-value-vector-reader" : "NO",
"range-index" : "NO"
},
}
}
```

## Exposing reload status as JMX Metrics

In addition to a global status, exposing the individual reload status as well as the global status would be very helpful in regards to configuration alerts on a prometheus stack.

Contributor guide

Open the contributing guide

Research direction

The issue identifies GET /segments/{tableName}_OFFLINE/metadata?type=OFFLINE as the current entry point for per-segment reload information; start by tracing that status and Pinot's JMX metric exposure. Done means JMX provides both a global table reload status and individual segment reload statuses suitable for operational alerts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.