apache / apache/uniffle

[FEATURE] Support local storage capacity selector for per-partition

Open
#845 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

Currently, the local storage of per-partition is selected by hash strategy. This is simple but will cause many problems, like many huge partitions will store in the same local disk, if the hash number is same. I know this is rare but it has happened in our internal env.

So it's better to support remaining space based storage selector for per-partition. And I will introduce the pluggable local storage selector, but I'm not sure whether it could support multiple disk storages for one-partition.

Anyway, the pluggable storage selector for localStorageManager is the first step to achieve above.

### Motivation

_No response_

### Describe the solution

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by locating localStorageManager and the current hash-based per-partition local-storage selection. Clarify how a pluggable selector should choose remaining capacity and whether one partition may use multiple disks; the work is done when the selector contract and supported behavior are specified and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.