Query-level priority and control

I want to open a discussion around how to add query-level priority and control. Though Pinot has tenants at the servers and brokers level, often we need finer-granular control at the query level.

In production, we are facing the following common challenges:
- Bad queries. We have seen some expensive queries, due to mistakes or sloppy ones from amateur users, could cause the scan of very large number of segments, resulting in high-GC, zookeeper timeouts, and eventually bringing down the servers in the entire tenant.
- Query priorities. Not all queries have the same priority. There are two main categories of queries we observe today: 1) fixed query periodically run, for example, to power dashboards; 2) ad-hoc queries to derive insights. Often, the latter category is less important but its workload is not predictable and therefore may breach the SLA of the first category.

I'd like to hear your thoughts on solving this type of challenge. While it's possible to explore a formal resource-based quota control framework at finer-granularity as other query engines do like Presto or Hive/Yarn, I'm also interested in some quick solutions to solve challenge #1, which can happen often and easily. For example, can we reject the expensive queries with the cost-estimate at the query plan phase? Or tools to identify expensive ongoing queries and be able to kill them?

No files, tests, or entry points are named. Begin by reviewing the proposed query-level priority, cost rejection, and ongoing-query termination approaches in Pinot; done would require an agreed design and implementation scope.

Written by the indexing model from the issue text.