apache / apache/pinot

Allow a server to host segments on multi storage tiers

Open
#8,843 4 comments 0 reactions 1 assignee Claimed by @klsince View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

As of today, [a table with multi-tiers](https://docs.pinot.apache.org/operators/operating-pinot/tiered-storage#tier-config) has to use different server pools to support multi storage tiers, and relies on SegmentRelocator to move the segments among those server pools to put them on the right storage tier. An example cluster setup may be like using a pool of servers with SSD as a tier and another pool of servers with HDD as a second tier.

But this model doesn't work when a server have both SSD and HDD, and wants to define multi tiers backed by them.

We'd like to address this issue. And one possible way is to implement a SegmentDirectoryLoader that's aware of multi data dirs; and enhance current way of tier migration to move segments across tiers even if they are not relocated to a different server, i.e. to be done on the same server locally.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.