Automattic / Automattic/hostmgr
Support uploading <5MB files to AWS
- Dominant language
- Swift
- Stars
- 12
- Forks
- 3
- Avg merge
- 4h 50m
- Merged PRs (30d)
- 4
Description
AWS Multipart upload seems to only be allowed if the file to upload is ≥5MB.
## Context
- 5MB is indeed the minimum size for each part of a multipart-upload.
- That minimum part size doesn't apply to the **_last_** part of a multipart upload (which makes sense, because the total file size is rarely an entire multiple of the part size, so there's always some remainder), so one could have thought that it would still be valid to use Multipart Upload for a <5MB file, by making it a multipart operation… with a single part (it thus being the last part and not subject to the minimum)
- But it seems that this is not the case, as we encountered an error when trying to upload a <5MB git cache archive in the past — see https://github.com/Automattic/hostmgr/pull/72
## Action Plan
- Double-check if the error we encountered during https://github.com/Automattic/hostmgr/pull/72 was indeed due to AWS not allowing Multipart Uploads for files <5MB (aka creating a multipart upload with only a single part), as opposed to a separate quirk in our implementation
- I doubt it, but since that would lead to a simple fix and avoid implementing extra stuff, worth checking anyway)
- If AWS indeed doesn't allow single-part MultipartUpload operations:
- Implement direct (aka non-Multipart) upload operation
- Implement the logic in tinys3 to choose the right operation (Multipart vs Direct) when asked to upload a file
- Note: the cutoff for the decision could (?) be greater than 5MB 🤔 maybe we don't need to use multipart-upload for files under 10MB? 🤷
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in the tinys3 upload logic and reproduce the failure described in the hostmgr PR #72 to determine whether AWS rejects a single-part multipart upload. Confirm the appropriate cutoff, then verify that uploads below it use a direct operation and larger files still use multipart upload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, swift
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100