Resume S3 MultipartUpload across PHP process boundaries via persisted UploadId
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 36.9k
- Forks
- 5.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 713
Description
[!TIP]
Help move this idea forward
- Use the 👍 reaction to show support for this feature.
- Avoid commenting unless you have relevant information to add; unnecessary comments create noise for subscribers.
- Subscribe to receive notifications about status changes and new comments.
Is your feature request related to a problem? Please describe.
\OC\Files\ObjectStore\S3ObjectTrait::writeMultiPart holds the Aws\Multipart\UploadState (containing the UploadId) as a local variable. The state is not persisted. When the PHP process terminates abnormally, the UploadId is lost.
The in-progress multipart upload remains on the S3 destination. Nextcloud has no reference to it. Subsequent MOVE/COPY/PUT attempts on the same source restart from part 0 and orphan the previous parts.
Describe the solution you'd like
Persist the UploadId, source-storage id, source internal path, and destination internal path at CreateMultipartUpload time. Store in oc_filecache_extended or an equivalent table.
On the next MOVE/COPY/PUT against the same source, writeMultiPart MUST look up the persisted UploadId. When present, reconstruct the UploadState via \Aws\S3\MultipartUploadingTrait::getStateFromService($client, $bucket, $key, $uploadId) and pass it to MultipartUploader via the state config option.
Delete the persisted record on CompleteMultipartUpload or AbortMultipartUpload.
Describe alternatives you've considered
- Retry from part 0 on the next attempt. Wastes upload bandwidth equal to the completed part count.
- Client-side chunked-upload resume (PR #27034). WebDAV Chunk-v2, client-driven, unrelated to server-side
MultipartUploader. - Same-request retry with reduced concurrency (PR #53580). Bounded to a single PHP process, does not survive process death.
- Orphan reap via scheduled
ListMultipartUploads+AbortMultipartUpload. Cleanup pattern only, forfeits progress.
Additional context
MultipartUploadingTrait::getStateFromServicereconstructsUploadStatefrombucket,key,uploadIdviaListPartspagination. Documented at https://docs.aws.amazon.com/aws-sdk-php/v3/api/class-Aws.S3.MultipartUploadingTrait.html.- Related open issues: #48823, #51860, #39088, #29841.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at OC\Files\ObjectStore\S3ObjectTrait::writeMultiPart and trace the CreateMultipartUpload, CompleteMultipartUpload, and AbortMultipartUpload paths. Review oc_filecache_extended or the equivalent persistence area, then examine Aws\S3\MultipartUploadingTrait::getStateFromService. Done means uploads can recover a persisted UploadId across PHP process boundaries and remove its record after completion or abort.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, php
- Domain
- backend, cloud, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100