apache / apache/uniffle

[FEATURE] Assignment strategies under different memory capacity configurations

Open
#1,079 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

If using the different memory capacity in one uniffle cluster and the underlying disk capacity is same, it will make the large memory capacity machine disk usage relatively high. Sometimes this will make local disk full and fallback to HDFS (it's really slow)

Anyway, I know if the HDFS is slow and local disk is full, this will reflect to the server memory of large memory machine. But this will also make some apps slow and the little memory machines are free when this happened.

I hope the better assignment strategies could be introduced for this problem. And I'm bothered by this. Feel free to describe more for this.

### Motivation

_No response_

### Describe the solution

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

The issue describes assignment strategies for clusters with different memory capacities and equal disk capacity, but names no files, tests, or entry points. Start by locating the existing assignment strategy implementation and its tests; done should include a defined strategy that balances disk usage without relying prematurely on HDFS.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.