[Umbrella] Object Storage Support (Help Wanted)
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues) and found no similar issues.
### Describe the proposal
Now, remote shuffle storage only support Hadoop Compatible Filesystem. Object storage is also important and widely used in the big data system. Object storage has different implement. Some systems don't support the method of `list` or have bad performance of the method `list`. Some systems need to consider file names to use more buckets of object. Some object storage don't support the method `append`. Now, we use store index and data separately, it will cause many small index files if we don't support the method `append`. So we should consider merging index files and data files. To have better performance, we should support object storage to adapt different object storage systems.
https://docs.google.com/document/d/1E88wZA9Yhr-pGeUEfxo6uSgsIXxg_ivPYBNcTOeaaZA/edit
### Task list
- [ ] https://github.com/apache/incubator-uniffle/issues/892
- [ ] https://github.com/apache/incubator-uniffle/issues/391
- [ ] documents for object storage (need to create an issue)
- [ ] https://github.com/apache/incubator-uniffle/issues/1133
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Contributor guide
Research direction
Read the linked issues 892, 391, and 1133 and the proposal document first. Identify the concrete implementation scope and missing documentation; this umbrella issue is done only when its listed object-storage work is completed and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100