new feature: support copying objects across buckets and regions.
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Currently, the `copy` API in OpenDAL can only copy an object within the same bucket.
In fact, object store services like [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) and [Tencent Cloud COS](https://www.tencentcloud.com/document/product/436/10881) already support copy objects across buckets and regions.
It will be very nice if OpenDAL can also support this feature.
### Problem and Solution
Take the COS implementation in OpenDAL as an example:
```rust
/// impl CosCore
pub async fn cos_copy_object(&self, from: &str, to: &str) -> Result> {
let source = build_abs_path(&self.root, from);
let target = build_abs_path(&self.root, to);
let source = format!("/{}/{}", self.bucket, percent_encode_path(&source));
let url = format!("{}/{}", self.endpoint, percent_encode_path(&target));
let mut req = Request::put(&url)
.header("x-cos-copy-source", &source)
.body(Buffer::new())
.map_err(new_request_build_error)?;
self.sign(&mut req).await?;
self.send(req).await
}
/// impl Access for CosBackend
async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result {
let resp = self.core.cos_copy_object(from, to).await?;
let status = resp.status();
match status {
StatusCode::OK => Ok(RpCopy::default()),
_ => Err(parse_error(resp)),
}
}
```
When constructing `target` and `url`, we can use difference buckets to fill the parameters instead of the same bucket. And the other bucket information can be put in `OpCopy`.
The request example in https://www.tencentcloud.com/document/product/436/10881:
```
PUT / HTTP/1.1
Host: .cos..myqcloud.com
Date: GMT Date
x-cos-copy-source: .cos..myqcloud.com/
Content-Length: 0
Authorization: Auth String
```
### Additional Context
_No response_
### Are you willing to contribute to the development of this feature?
- [x] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
Start with OpenDAL's copy API and the OpCopy parameters, then compare the COS implementation with the AWS S3 and Tencent Cloud COS request documentation linked in the issue. Define how source and target bucket and region information should be represented, then verify that cross-bucket and cross-region copies are supported without breaking same-bucket copies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100