apache / apache/pinot

Using deep storage as a tiered storage

Open
#5,554 2 comments 0 reactions 1 assignee Claimed by @kishoreg View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Currently IIUC all the segments need to be on disk to be able to query them.
Usually, queries to older data tends to be lower or clients would be okay with
in-curing latency for queries on older data.

Using this assumption, we could possibly keep the segments older than X days
on deep storage and not load them onto the servers.

Couple of ways to deal with this based on discussions on slack:
1) Using presto on top of pinto and make presto-pinot be able to query
segments on S3 directly.
2) When the query arrives lazy load the segment and return the results.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.