source-cooperative / source-cooperative/data.source.coop
Default new uploads to S3 Intelligent-Tiering storage class
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 24
- Forks
- 6
- Avg merge
- 1h 32m
- Merged PRs (30d)
- 1
Description
Proposed change
When the CLI uploads objects to S3, set StorageClass: INTELLIGENT_TIERING by default on:
PutObjectrequestsCreateMultipartUploadrequests (this is where storage class is set for multipart — not on individualUploadPartcalls; easy to miss in review)
Add a --storage-class flag (and corresponding env var, e.g. SOURCE_COOP_STORAGE_CLASS) that lets users override the default. Default value: INTELLIGENT_TIERING.
--storage-class is the standard AWS CLI flag name on s3 cp / s3 sync and the standard StorageClass field in the AWS SDKs, so plumbing through should be straightforward.
Accepted values (validate against this allowlist; reject others with a clear error):
| Value | Use case |
|---|---|
INTELLIGENT_TIERING |
Default — auto-tiering, millisecond retrieval, no retrieval fees on default tiers |
STANDARD |
Override for users who explicitly want hot storage |
Glacier-tier classes (GLACIER_IR, GLACIER, DEEP_ARCHIVE) are intentionally not supported in this initial release. They have meaningful operational implications — retrieval fees, retrieval latency, separate user expectations for cold data — that we want to design proper support around (UX for restore requests, documentation, possibly publisher-facing tooling) before exposing them. We'll add them in a follow-up once that work is scoped.
Other classes (ONEZONE_IA, STANDARD_IA, REDUCED_REDUNDANCY, EXPRESS_ONEZONE, OUTPOSTS, SNOW) are out of scope: either inappropriate for a durable public archive, deprecated, or not applicable to Source's deployment.
Acceptance criteria
- Single-part uploads default to
INTELLIGENT_TIERING. - Multipart uploads default to
INTELLIGENT_TIERING(set onCreateMultipartUpload). -
--storage-classflag overrides the default and accepts the standard S3 storage class names. - Behavior is documented in the CLI's README /
--helpoutput, including the rationale (cost) so users understand what the default does. - Released and available via the standard install path (homebrew tap + GitHub releases) by April 30, 2026.
Why the date
We want May 2026 to be a clean month of billing data that reflects the optimized state. With both the bucket lifecycle rule and the CLI default in place by April 30, the May AWS bill will be the first full month under the new configuration, which gives us a clean before/after comparison against the Nov 2025 – Apr 2026 baseline.
Notes / things to verify
- Confirm whether the CLI uses the AWS SDK or shells out to
aws s3 cp/aws s3 sync. If it shells out, plumbing--storage-classthrough is sufficient. If it uses the SDK directly, set on the relevant request structs. - The 30-day minimum storage charge for Intelligent-Tiering means objects deleted within 30 days of upload still incur the 30-day charge. This is essentially a non-issue for a public archive but worth a one-line note in the docs.
- We are not adding a bucket policy that requires the storage class header. Web UI uploads, browser-based uploads via presigned URL, and stock-SDK clients should continue to work; the lifecycle rule will catch them server-side.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the CLI upload entry points and checking whether they use AWS SDK PutObject/CreateMultipartUpload calls or invoke aws s3 commands. Trace the storage-class flag and environment configuration, then verify single-part and multipart behavior, accepted-value errors, README/help documentation, and the standard release path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cli, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100