tracking: implement native Writer::copy_from for additional services
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
## Goal
Track native service implementations of `Writer::copy_from` after the portable API, streaming fallback, and S3 `UploadPartCopy` implementation landed in #8140.
`Writer::copy_from` is available on every service with read and write support. A service sets `Capability::write_can_copy_from` only when its writer can insert every supported `OpRead` input into the existing destination transaction while preserving range, version, conditions, ordering, completion, and abort semantics. Otherwise OpenDAL keeps using the bounded streaming fallback.
## Completed
- [x] Land the public API, composed streaming fallback, multipart scheduling, layer forwarding rules, and capability in #8140.
- [x] Implement S3 `UploadPartCopy` in the writer's multipart upload transaction in #8140.
- [ ] Validate the S3-compatible provider presets that share `services/s3`; add provider-specific capability gating if an actual provider does not support the required `UploadPartCopy` contract.
## High-confidence multipart candidates
These services already use `oio::MultipartWriter`, and their storage APIs can copy a bounded source range into an existing multipart or large-file transaction.
- [ ] **TOS** — Reuse the existing `TosUploadPartCopyRequest` and `TosCopier` implementation from `MultipartWrite::copy_part`. Extend the request to preserve source version and all four advertised read conditions before enabling `write_can_copy_from`. TOS documents versioned `UploadPartCopy` and the copy-source condition headers.
- [ ] **OSS** — Add `UploadPartCopy` to `OssWriter`. OSS supports bounded source ranges, `versionId`, and all four copy-source conditions. The existing multipart writer already owns initiation, ordered completion, and abort. [UploadPartCopy API](https://www.alibabacloud.com/help/en/oss/developer-reference/uploadpartcopy)
- [ ] **COS** — Add Upload Part Copy to `CosWriter`, including source version and all four advertised read conditions. The existing multipart writer can reuse the same upload ID and part sequence for local and copied inputs. [Upload Part Copy API](https://www.tencentcloud.com/document/product/436/8287)
- [ ] **OBS** — Add Copy Part to `ObsWriter`, including all four advertised read conditions. OBS supports source ranges and conditional copy headers inside a multipart upload. [Copying Parts API](https://support.huaweicloud.com/intl/en-us/tr-central-201-api-obs/obs_04_0100.html)
- [ ] **Backblaze B2** — Resolve the source path to a file ID and call `b2_copy_part` with the writer's large-file ID, part number, and range. Update part checksum handling so uploaded and copied parts produce the SHA-1 list expected by `b2_finish_large_file`. B2 currently advertises no conditional or versioned reads, so no additional `OpRead` lowering is required. [`b2_copy_part`](https://www.backblaze.com/apidocs/b2-copy-part)
- [ ] **Swift** — Server-side copy each source range into the SLO segment path with `PUT` + `X-Copy-From` + `Range`, then reuse the existing SLO manifest completion and cleanup. Verify the advertised source conditions against an actual Swift deployment before enabling the capability. [Object copy API](https://docs.openstack.org/api-ref/object-store/index.html)
## Block-writer prerequisite
- [ ] Extend `oio::BlockWrite` and `oio::BlockWriter` with ordered native copy blocks under the same concurrency, completion, and abort state used by local blocks.
- [ ] **Azure Blob** — Implement `PutBlockFromURL` through that block-writer path. Reuse the existing block copier request, preserve blob version and all four source conditions, and commit copied and uploaded block IDs through the writer's existing `Put Block List` transaction. [Put Block From URL API](https://learn.microsoft.com/en-us/rest/api/storageservices/put-block-from-url)
## Investigation
- [ ] **GCS** — Test whether the actual XML API accepts `x-goog-copy-source` together with `uploadId` and `partNumber`. The current multipart documentation describes uploaded data parts but does not document UploadPartCopy, and it states that multipart preconditions are unsupported. Do not enable `write_can_copy_from` based only on S3 compatibility. [XML API multipart uploads](https://cloud.google.com/storage/docs/multipart-uploads)
- [ ] **Azure Files** — `Put Range From URL` can copy a source range, but the current service uses `OneShotWriter`/`AppendWriter`, and file creation requires the destination size before range writes. Define a writer transaction that satisfies online scheduling and abort semantics before treating it as an implementation candidate. [Put Range From URL API](https://learn.microsoft.com/en-us/rest/api/storageservices/put-range-from-url)
Services that only expose whole-object copy, rename, rewrite, or compose remain on the portable streaming path unless their API can place arbitrary bounded ranges into the same open writer transaction. A whole-object `copy()` implementation alone is not sufficient.
## Acceptance checklist for every service PR
- [ ] Reproduce and verify the native operation against the actual service before changing service code or capability advertisement.
- [ ] Preserve every `OpRead` option advertised by the service, including source version and conditions; otherwise leave the capability false.
- [ ] Keep local uploads and native copied ranges in one ordered writer transaction and one chunk/concurrency budget.
- [ ] Preserve arbitrary `write`/`copy_from` interleaving, multipart minimum sizes, and the 5 GiB public per-call limit.
- [ ] Treat every native execution error as terminal for the writer while keeping `abort` available; do not retry through streaming fallback after mutation may have occurred.
- [ ] Verify `close`, `abort`, range boundaries, condition failures, versions where supported, and a mixed local/native byte sequence.
- [ ] Disable the capability for incompatible write configurations such as checksums or encryption modes that native copied parts cannot satisfy.
- [ ] Document service-specific permissions, region/account constraints, encryption limitations, and observable fallback cases.
## References
- RFC: #8109
- Core and S3 implementation: #8140
## AI assistance
This issue update was drafted with assistance from OpenAI Codex. The candidate list is based on the merged RFC, current OpenDAL service implementations, and linked provider documentation. Every service task still requires actual-service verification under the repository's contribution policy.
Contributor guide
Research direction
Start with the core and S3 implementation in #8140, then inspect the relevant service writer entry point, such as MultipartWriter, BlockWriter, or a listed service writer. Choose one unchecked service and verify its native API against the stated read conditions and transaction semantics. Done means the service is tested against the actual provider, preserves mixed writes and copies, handles completion and abort, and advertises the capability only when valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100