Create a dashboard for most expensive queries per table

In many production settings, a Pinot cluster can handle thousand of queries per second. Expensive queries can greatly impact the overall system performance. Right now, users of Pinot have to rely on external mechanisms of pinpoint these costly queries. Namely, one has to export the broker query logs with query stats and then rely on tools like Kibana to filter and find those queries.

We propose to add query monitoring to Pinot controller console. The query monitoring console should show the most expensive (according to query stats like number of docs/entries scanned) queries per table in the past N days. One can use the dashboard to:

1. Find out the most expensive queries per table.
2. Help to find missing index config for the corresponding table.
3. (Nice to have) allow users to create the indexes to improve query performance.

@icefury71 @Jackie-Jiang

Start by reviewing the Pinot controller console and the broker query logs with their query stats, then compare how Kibana currently identifies costly queries. Done means the console presents the most expensive queries per table over a selectable past-N-day period; index creation is explicitly marked as a nice-to-have.

Written by the indexing model from the issue text.