[Improvement] HDD friendly deployment
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
### Motivation
Due to Uniffle's multi-storage design, it's crucial that the second storages perform well.

Currently, the recommend deploy choices are `MEMORY_LOCALFILE`, `MEMORY_HDFS` and `MEMORY_LOCALFILE_HDFS`, and such deployments are used in production in `iQiyi`, `Didi`, `Tencent`, etc.
For `MEMORY_HDFS` deployment, it might be suitable for large shuffle. Due to HDFS' I/O model, it may not handle small shuffles very well. Therefore, `MEMORY_LOCALFILE` and `MEMORY_LOCALFILE_HDFS` are the de facto choices to cover most shuffle scenarios.
The storage choice of `LOCALFILE` could be traditional HDD disks, or SSD or higher-speed NVMe SSDs. The SSD storages are preferred. However due to its cost and hardware constraints, SSD cannot be deployed widely.
Therefore, I'd like to propose some improvements to Uniffle's HDD deployment, make it more widely applicable.
### How should we improve?
1. Reduce small I/Os of local storages:
- The flush strategy of shuffle server could be improved, to wait for more shuffle data before flush to disk.
- Leverages range partition to merge multiple partition's shuffle data in one file.
- Defer flush of index file, which normally causes a small I/O.
2. I/O capability report, the shuffle worker shall report the local storage's hard drive type: HDD, SSD, etc and the disk numbers.
3. Better balancing strategy in coordinate and shuffle worker side:
- coordinate side: partition size/throughput aware assignment to balancing workload in shuffle workers.
- shuffle server side: distributes workload evenly to leverages multiple HDD disks. A new disk selection strategy should be deployed and I/O stats aware. cc @zuston
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Contributor guide
Research direction
No files, tests, or concrete entry points are identified. First read the shuffle server storage and flush paths, then the coordinator and shuffle-worker assignment logic, including local disk capability and I/O-stat reporting; before coding, split the proposal into a scoped improvement with measurable completion criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100