Using deep storage as a tiered storage
Open
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Currently IIUC all the segments need to be on disk to be able to query them.
Usually, queries to older data tends to be lower or clients would be okay with
in-curing latency for queries on older data.
Using this assumption, we could possibly keep the segments older than X days
on deep storage and not load them onto the servers.
Couple of ways to deal with this based on discussions on slack:
1) Using presto on top of pinto and make presto-pinot be able to query
segments on S3 directly.
2) When the query arrives lazy load the segment and return the results.
Contributor guide
Assessment
This issue has not been assessed yet.