Automattic / Automattic/hostmgr

Support uploading <5MB files to AWS

Open
#95 0 comments 0 reactions 0 assignees View on GitHub
bug hostmgr
Dominant language
Swift
Stars
12
Forks
3
Avg merge
4h 50m
Merged PRs (30d)
4

Description

AWS Multipart upload seems to only be allowed if the file to upload is ≥5MB.

## Context

- 5MB is indeed the minimum size for each part of a multipart-upload.
- That minimum part size doesn't apply to the **_last_** part of a multipart upload (which makes sense, because the total file size is rarely an entire multiple of the part size, so there's always some remainder), so one could have thought that it would still be valid to use Multipart Upload for a <5MB file, by making it a multipart operation… with a single part (it thus being the last part and not subject to the minimum)
- But it seems that this is not the case, as we encountered an error when trying to upload a <5MB git cache archive in the past — see https://github.com/Automattic/hostmgr/pull/72

## Action Plan

- Double-check if the error we encountered during https://github.com/Automattic/hostmgr/pull/72 was indeed due to AWS not allowing Multipart Uploads for files <5MB (aka creating a multipart upload with only a single part), as opposed to a separate quirk in our implementation
- I doubt it, but since that would lead to a simple fix and avoid implementing extra stuff, worth checking anyway)
- If AWS indeed doesn't allow single-part MultipartUpload operations:
- Implement direct (aka non-Multipart) upload operation
- Implement the logic in tinys3 to choose the right operation (Multipart vs Direct) when asked to upload a file
- Note: the cutoff for the decision could (?) be greater than 5MB 🤔 maybe we don't need to use multipart-upload for files under 10MB? 🤷

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the tinys3 upload logic and reproduce the failure described in the hostmgr PR #72 to determine whether AWS rejects a single-part multipart upload. Confirm the appropriate cutoff, then verify that uploads below it use a direct operation and larger files still use multipart upload.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, swift
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.