nextcloud / nextcloud/server

Resume S3 MultipartUpload across PHP process boundaries via persisted UploadId

Open
#62,649 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage enhancement feature: object storage hotspot: file transfer performance performance 🚀
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

[!TIP]

Help move this idea forward
  • Use the 👍 reaction to show support for this feature.
  • Avoid commenting unless you have relevant information to add; unnecessary comments create noise for subscribers.
  • Subscribe to receive notifications about status changes and new comments.

Is your feature request related to a problem? Please describe.

\OC\Files\ObjectStore\S3ObjectTrait::writeMultiPart holds the Aws\Multipart\UploadState (containing the UploadId) as a local variable. The state is not persisted. When the PHP process terminates abnormally, the UploadId is lost.

The in-progress multipart upload remains on the S3 destination. Nextcloud has no reference to it. Subsequent MOVE/COPY/PUT attempts on the same source restart from part 0 and orphan the previous parts.

Describe the solution you'd like

Persist the UploadId, source-storage id, source internal path, and destination internal path at CreateMultipartUpload time. Store in oc_filecache_extended or an equivalent table.

On the next MOVE/COPY/PUT against the same source, writeMultiPart MUST look up the persisted UploadId. When present, reconstruct the UploadState via \Aws\S3\MultipartUploadingTrait::getStateFromService($client, $bucket, $key, $uploadId) and pass it to MultipartUploader via the state config option.

Delete the persisted record on CompleteMultipartUpload or AbortMultipartUpload.

Describe alternatives you've considered

  • Retry from part 0 on the next attempt. Wastes upload bandwidth equal to the completed part count.
  • Client-side chunked-upload resume (PR #27034). WebDAV Chunk-v2, client-driven, unrelated to server-side MultipartUploader.
  • Same-request retry with reduced concurrency (PR #53580). Bounded to a single PHP process, does not survive process death.
  • Orphan reap via scheduled ListMultipartUploads + AbortMultipartUpload. Cleanup pattern only, forfeits progress.

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at OC\Files\ObjectStore\S3ObjectTrait::writeMultiPart and trace the CreateMultipartUpload, CompleteMultipartUpload, and AbortMultipartUpload paths. Review oc_filecache_extended or the equivalent persistence area, then examine Aws\S3\MultipartUploadingTrait::getStateFromService. Done means uploads can recover a persisted UploadId across PHP process boundaries and remove its record after completion or abort.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, php
Domain
backend, cloud, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.