Pinot Long Term Data Store

We have certain use-cases wherein we would like to move older data to cheap object stores like the S3 and store only the most recent data in Pinot. One such use-case is storing distributed trace data - Our query patterns show that more than 90% of the queries lie in the last 24 hours. Having said that, we have a retention period of 30 days. So in this case, we would like to keep only the last 24 hours worth of data in Pinot, and move the rest to a cheap store like the S3.

From what I concluded from our initial discussions, this would involve work on two fronts:

1. Moving older data from Pinot to the S3 (this already happens, mentioning it for completeness).
2. Pinot-Presto connector, so that it can query from both Pinot and the S3 based on the time span of the query.

Here is a rough sketch of the design we came up with in our discussion the other day (this is relevant to the distributed tracing example I gave above):


I have created this issue to get the discussion started.

Thanks!

No files or tests are named. Start by reviewing the Pinot-Presto connector and the existing S3 data-movement path, then clarify how queries should combine recent Pinot data with older S3 data; done requires an agreed design and implementation scope.

Written by the indexing model from the issue text.