nextcloud / nextcloud/server

files_external amazons3: cross-mount MOVE should use server-side MultipartCopy when peer is same S3 endpoint

Open
#62,645 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage enhancement feature: external storage feature: object storage performance 🚀
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

[!TIP]

Help move this idea forward
  • Use the 👍 reaction to show support for this feature.
  • Avoid commenting unless you have relevant information to add; unnecessary comments create noise for subscribers.
  • Subscribe to receive notifications about status changes and new comments.

Is your feature request related to a problem? Please describe.

\OC\Files\Storage\Common::copyFromStorage routes every cross-mount MOVE through PHP. When both mounts are files_external amazons3 on the same S3 endpoint, the source is read via fopen('r') and re-uploaded via \Aws\S3\MultipartUploader. The uploader's non-seekable branch materialises each part into php://temp before dispatch, so every part transits local temp storage. Temp usage scales with object size.

If the destination rejects a part, \Aws\Multipart\AbstractUploadManager::getResultHandler records the error and continues yielding parts. Temp usage grows unbounded.

Describe the solution you'd like

Add a fast path in Common::moveFromStorage (or an \OCA\Files_External\Lib\Storage\AmazonS3::copyFromStorage override). When source and destination are both AmazonS3 on the same endpoint, dispatch \Aws\S3\MultipartCopy (server-side UploadPartCopy with x-amz-copy-source).

Bytes stay on the S3 side. No PHP staging. Wall-clock reduces to seconds regardless of object size.

Precondition: the destination tenant's principal must hold s3:GetObject on the source bucket.

Describe alternatives you've considered

  • Lower uploadPartSize and concurrency on the mount. Shrinks per-part footprint. Keeps O(N) buffering.
  • Bucket-side LifecycleConfiguration AbortIncompleteMultipartUpload. Cleans up after the fact. Unsupported by NetApp StorageGrid.
  • Out-of-band aws s3 copy. Bypasses Nextcloud. oc_filecache stays stale.
  • #47986 (merged, NC 31). Same-bucket fast path inside ObjectStoreStorage. Cross-mount files_external remains uncovered.

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Common::moveFromStorage and the files_external amazons3 storage implementation, then compare the same-bucket fast path from issue #47986. Confirm how source and destination mounts expose their S3 endpoints and how MultipartCopy is invoked. Done means same-endpoint cross-mount moves use server-side multipart copying without PHP temp staging while retaining Nextcloud file-cache behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, php
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.