apache / apache/uniffle

[Umbrella] Object Storage Support (Help Wanted)

Open
#1,030 16 comments 1 reaction 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues) and found no similar issues.

### Describe the proposal

Now, remote shuffle storage only support Hadoop Compatible Filesystem. Object storage is also important and widely used in the big data system. Object storage has different implement. Some systems don't support the method of `list` or have bad performance of the method `list`. Some systems need to consider file names to use more buckets of object. Some object storage don't support the method `append`. Now, we use store index and data separately, it will cause many small index files if we don't support the method `append`. So we should consider merging index files and data files. To have better performance, we should support object storage to adapt different object storage systems.
https://docs.google.com/document/d/1E88wZA9Yhr-pGeUEfxo6uSgsIXxg_ivPYBNcTOeaaZA/edit

### Task list

- [ ] https://github.com/apache/incubator-uniffle/issues/892
- [ ] https://github.com/apache/incubator-uniffle/issues/391
- [ ] documents for object storage (need to create an issue)
- [ ] https://github.com/apache/incubator-uniffle/issues/1133

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Read the linked issues 892, 391, and 1133 and the proposal document first. Identify the concrete implementation scope and missing documentation; this umbrella issue is done only when its listed object-storage work is completed and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.