new feature: unify copy and rename's atomicity semantic
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Originated from Xuanwo's [comment](https://github.com/apache/opendal/pull/7535#issuecomment-4795815743) in #7535 .
OpenDAL's `copy` and `rename` has different atomicity depends on the backend support. For example, `fs` backend will attempt `rename` and set metadata. This behavior is similar to `cp --preserve=xattr`. But the two operations in rename are not atomic. (Nor `cp --preserve=xattr` is atmoic.)
A common trick for applications is to copy a file to a temporary location and then set xattr. And then move it to a destination. This approach is obviously limited in file system.
### Problem and Solution
For OpenDAL, we might need a proper solution to define:
1. Principle: Do we want to guarantee atomicity for `copy` and `rename?
2. API design: How do we define API for atomic operation?
## Principle: Do we want to guarantee atomicity for `copy` and `rename?
Some notable implementations:
- s3 has strongly consistence for metadata operations along with objects.
- Linux doesn't set xattr with files in an atomic operation. We might be able to simulate that but this will result in a performance lost. Some file systems doesn't support copy on write.
## API design
1. Do we want to have `copy` and `rename` to be atomic by default?
2. If impossible, do we add an atomic option?
3. Along with 2, we could have an capability hinting if using atomic property will result in performance loss.
### Additional Context
- OpenDAL's current behavior depends on backend implementations.
- S3 allows copy or replace metadata by [x-amz-object-annotation-directive](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html#API_CopyObject_RequestSyntax) when copy an object.
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing the current copy and rename implementations across backends, along with the capability definitions; the issue is done when the atomicity principle and a concrete API design are agreed and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100