compiler-explorer / compiler-explorer/infra
Discovery uploads hardcode REDUCED_REDUNDANCY, which costs more than STANDARD and is far less durable
- Dominant language
- Python
- Stars
- 434
- Forks
- 430
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 38
Description
> Written by Claude (an LLM), at Matt's request. Companion to compiler-explorer/compiler-explorer#9020, which covers the same mistake in the app's `S3Cache`.
Four places in this repo hardcode the Reduced Redundancy storage class when uploading **compiler discovery** output:
```
bin/lib/discovery.py:52 s3_config = {"ACL": "public-read", "StorageClass": "REDUCED_REDUNDANCY"}
bin/lib/cli/runner.py:61 _S3_CONFIG = dict(ACL="public-read", StorageClass="REDUCED_REDUNDANCY")
bin/lib/cli/win_runner.py:19 _S3_CONFIG = dict(ACL="public-read", StorageClass="REDUCED_REDUNDANCY")
bin/lib/cli/gpu_runner.py:16 _S3_CONFIG = dict(ACL="public-read", StorageClass="REDUCED_REDUNDANCY")
```
RRS is a legacy class that is now **more expensive** than Standard: $0.02400 vs $0.02300 per GB-month in us-east-1 (AWS price list `20260807185915`, the figures verified for #9020).
### Why this one is worth fixing on durability, not price
The money is trivial and I would rather say so up front. `dist/discovery/` is **163 GiB across 1,493 objects**, so the premium is about **$0.16/month**.
The durability argument is the real one. RRS is designed for 99.99% durability, an average expected loss of 0.01% of objects per year, against Standard's eleven nines. At 1,493 objects that is roughly **0.15 objects lost per year in expectation**, and the store already goes back to 2021-10-16. Unlike cache entries these are **release-pinned artifacts**: a discovery file is generated once per build and read at prod startup, and regenerating an old one is not always possible because it reflects the compilers installed at the time. So we are paying a premium for materially lower durability on data that is less reconstructible than the cache the same mistake was made on.
### Fix
Drop `StorageClass` from all four dicts and let it default to Standard. Keep `ACL="public-read"`; that is load-bearing. Only affects newly written objects; existing ones keep their class.
### Retention is a separate matter
The fact that `dist/discovery/` has no lifecycle rule at all is split out into #2322. This issue is only about the storage class, which is a one-line change per call site and independent of whatever retention window we land on.
Contributor guide
Research direction
Read the four named upload configurations in bin/lib/discovery.py, bin/lib/cli/runner.py, bin/lib/cli/win_runner.py, and bin/lib/cli/gpu_runner.py. Remove the hardcoded storage class while preserving ACL="public-read" at each call site, then verify the resulting uploads use the default Standard class. Retention and existing objects are outside this issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100