apache / apache/opendal

new feature: support copying objects across buckets and regions.

Open
#5,672 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Currently, the `copy` API in OpenDAL can only copy an object within the same bucket.

In fact, object store services like [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) and [Tencent Cloud COS](https://www.tencentcloud.com/document/product/436/10881) already support copy objects across buckets and regions.

It will be very nice if OpenDAL can also support this feature.

### Problem and Solution

Take the COS implementation in OpenDAL as an example:

```rust
/// impl CosCore
pub async fn cos_copy_object(&self, from: &str, to: &str) -> Result> {
let source = build_abs_path(&self.root, from);
let target = build_abs_path(&self.root, to);

let source = format!("/{}/{}", self.bucket, percent_encode_path(&source));
let url = format!("{}/{}", self.endpoint, percent_encode_path(&target));

let mut req = Request::put(&url)
.header("x-cos-copy-source", &source)
.body(Buffer::new())
.map_err(new_request_build_error)?;

self.sign(&mut req).await?;

self.send(req).await
}

/// impl Access for CosBackend
async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result {
let resp = self.core.cos_copy_object(from, to).await?;

let status = resp.status();

match status {
StatusCode::OK => Ok(RpCopy::default()),
_ => Err(parse_error(resp)),
}
}
```

When constructing `target` and `url`, we can use difference buckets to fill the parameters instead of the same bucket. And the other bucket information can be put in `OpCopy`.

The request example in https://www.tencentcloud.com/document/product/436/10881:

```
PUT / HTTP/1.1
Host: .cos..myqcloud.com
Date: GMT Date
x-cos-copy-source: .cos..myqcloud.com/
Content-Length: 0
Authorization: Auth String
```

### Additional Context

_No response_

### Are you willing to contribute to the development of this feature?

- [x] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start with OpenDAL's copy API and the OpCopy parameters, then compare the COS implementation with the AWS S3 and Tencent Cloud COS request documentation linked in the issue. Define how source and target bucket and region information should be represented, then verify that cross-bucket and cross-region copies are supported without breaking same-bucket copies.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.