files_external amazons3: cross-mount MOVE should use server-side MultipartCopy when peer is same S3 endpoint
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 36.9k
- Forks
- 5.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 713
Description
[!TIP]
Help move this idea forward
- Use the 👍 reaction to show support for this feature.
- Avoid commenting unless you have relevant information to add; unnecessary comments create noise for subscribers.
- Subscribe to receive notifications about status changes and new comments.
Is your feature request related to a problem? Please describe.
\OC\Files\Storage\Common::copyFromStorage routes every cross-mount MOVE through PHP. When both mounts are files_external amazons3 on the same S3 endpoint, the source is read via fopen('r') and re-uploaded via \Aws\S3\MultipartUploader. The uploader's non-seekable branch materialises each part into php://temp before dispatch, so every part transits local temp storage. Temp usage scales with object size.
If the destination rejects a part, \Aws\Multipart\AbstractUploadManager::getResultHandler records the error and continues yielding parts. Temp usage grows unbounded.
Describe the solution you'd like
Add a fast path in Common::moveFromStorage (or an \OCA\Files_External\Lib\Storage\AmazonS3::copyFromStorage override). When source and destination are both AmazonS3 on the same endpoint, dispatch \Aws\S3\MultipartCopy (server-side UploadPartCopy with x-amz-copy-source).
Bytes stay on the S3 side. No PHP staging. Wall-clock reduces to seconds regardless of object size.
Precondition: the destination tenant's principal must hold s3:GetObject on the source bucket.
Describe alternatives you've considered
- Lower
uploadPartSizeandconcurrencyon the mount. Shrinks per-part footprint. Keeps O(N) buffering. - Bucket-side
LifecycleConfiguration AbortIncompleteMultipartUpload. Cleans up after the fact. Unsupported by NetApp StorageGrid. - Out-of-band
aws s3copy. Bypasses Nextcloud.oc_filecachestays stale. - #47986 (merged, NC 31). Same-bucket fast path inside
ObjectStoreStorage. Cross-mountfiles_externalremains uncovered.
Additional context
\Aws\S3\MultipartCopy: https://docs.aws.amazon.com/aws-sdk-php/v3/api/class-Aws.S3.MultipartCopy.htmlUploadPartCopyIAM: https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html- Related open issues: #29841, #48194, #47856, #24540.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Common::moveFromStorage and the files_external amazons3 storage implementation, then compare the same-bucket fast path from issue #47986. Confirm how source and destination mounts expose their S3 endpoints and how MultipartCopy is invoked. Done means same-endpoint cross-mount moves use server-side multipart copying without PHP temp staging while retaining Nextcloud file-cache behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, php
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100