apache / apache/uniffle

[FEATURE] Expect to support the filesystem not implementing the append-mode.

Open
#391 16 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

Cloud object storage and its corresponding accelerated cache are widely used in storage-computing separation architecture.
But many of them may not implement the append-mode or the write amplification of the append-mode will be tricky.

Is it possible to consider supporting the way of independently storing each block as a data file at the same time?

### Motivation

Cloud object storage and its corresponding accelerated cache are widely used in storage-computing separation architecture.
But many of them may not implement the append-mode or the write amplification of the append-mode will be tricky.

### Describe the solution

The initially envisaged solution is to implement a non-append abstract storage type, such as: `AbstractObjectStorageWriteHandler`, `AbstractObjectStorageReadHandler`, and `AbstractObjectDeleteHandler`, and Implement the basic interface required by RSS in it.

The implementation of the `read`, `write`, and `delete` interfaces placed in the specific storage layer can be implemented by subclasses.

### Additional context

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing RSS storage interfaces and append-mode implementations, then compare them with the proposed AbstractObjectStorageWriteHandler, AbstractObjectStorageReadHandler, and AbstractObjectDeleteHandler. Define how read, write, and delete would operate for independently stored block files, and verify that a concrete storage layer can support this without append-mode requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.