awslabs / awslabs/filemoverexpress
[Feature] Resume individual S3 object transfers from where they left off
- Dominant language
- TypeScript
- Stars
- 4
- Forks
- 1
- Avg merge
- 9h 54m
- Merged PRs (30d)
- 41
Description
**Background**
File Mover Express supports pausing and resuming transfer jobs today. When a job is paused, tasks that haven't started are re-queued and will pick up on resume. However, any S3 object that was actively being uploaded at the time of the pause is discarded, its progress is lost and it starts over from the beginning when the job resumes.
This is particularly noticeable with large files. A 50GB file that was 90% uploaded when paused will restart from 0% on resume.
**Current behavior**
When a job is paused:
- The upload context is cancelled, stopping the active upload
- The task's BytesTransferred is reset to 0
- On resume, the task is re-queued as if it had never started
- The incomplete multipart upload is left orphaned in S3 (no AbortMultipartUpload is called)
- A brand new multipart upload is started from scratch
**Desired behavior**
When a job is paused and then resumed, an in-progress S3 object upload should resume from where it left off rather than restarting. If resuming from the exact byte offset is not feasible, it should at minimum resume from the last completed multipart part boundary.
**What needs to happen**
The core challenge is that the AWS SDK v2 manager.Uploader handles multipart uploads internally and does not expose the upload ID or completed parts. Implementing object-level resume will require:
- Tracking the multipart upload ID and completed parts per task — the Task struct currently has no fields for this
- On pause, calling AbortMultipartUpload OR storing the upload ID and completed parts so the upload can be continued later
- On resume, either continuing the existing multipart upload using UploadPart directly (bypassing the SDK manager), or at minimum aborting cleanly and restarting
- Handling the case where the multipart upload has expired in S3 (uploads expire after a configurable period)
**Relevant files**
pause-job.go - sets job/task status and cancels context
resume-job.go -re-queues paused tasks
transfer-worker.go - handles ErrJobPaused, resets BytesTransferred
upload.go - wraps manager.Uploader
task.go - Task struct, would need new fields for upload ID / part tracking
**Notes**
- Incomplete multipart uploads currently accumulate in S3. Until this is fixed, users should configure an S3 lifecycle rule to abort incomplete multipart uploads to avoid unnecessary storage costs. See AWS docs.
- The rename operation in rename-objects.go already has a working example of manual multipart upload handling with AbortMultipartUpload — this could serve as a reference.
Contributor guide
Research direction
Start by reading pause-job.go, resume-job.go, transfer-worker.go, upload.go, and task.go to trace cancellation, re-queuing, and progress reset; inspect rename-objects.go for the existing manual multipart-upload reference. The work is done when paused active S3 uploads preserve usable multipart progress or are cleanly handled, resume without unnecessarily restarting completed parts, and expired uploads are addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100