compiler-explorer / compiler-explorer/infra

Discovery uploads hardcode REDUCED_REDUNDANCY, which costs more than STANDARD and is far less durable

Open Beginner friendly
#2,321 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
434
Forks
430
Avg merge
4d 21h
Merged PRs (30d)
38

Description

> Written by Claude (an LLM), at Matt's request. Companion to compiler-explorer/compiler-explorer#9020, which covers the same mistake in the app's `S3Cache`.

Four places in this repo hardcode the Reduced Redundancy storage class when uploading **compiler discovery** output:

```
bin/lib/discovery.py:52 s3_config = {"ACL": "public-read", "StorageClass": "REDUCED_REDUNDANCY"}
bin/lib/cli/runner.py:61 _S3_CONFIG = dict(ACL="public-read", StorageClass="REDUCED_REDUNDANCY")
bin/lib/cli/win_runner.py:19 _S3_CONFIG = dict(ACL="public-read", StorageClass="REDUCED_REDUNDANCY")
bin/lib/cli/gpu_runner.py:16 _S3_CONFIG = dict(ACL="public-read", StorageClass="REDUCED_REDUNDANCY")
```

RRS is a legacy class that is now **more expensive** than Standard: $0.02400 vs $0.02300 per GB-month in us-east-1 (AWS price list `20260807185915`, the figures verified for #9020).

### Why this one is worth fixing on durability, not price

The money is trivial and I would rather say so up front. `dist/discovery/` is **163 GiB across 1,493 objects**, so the premium is about **$0.16/month**.

The durability argument is the real one. RRS is designed for 99.99% durability, an average expected loss of 0.01% of objects per year, against Standard's eleven nines. At 1,493 objects that is roughly **0.15 objects lost per year in expectation**, and the store already goes back to 2021-10-16. Unlike cache entries these are **release-pinned artifacts**: a discovery file is generated once per build and read at prod startup, and regenerating an old one is not always possible because it reflects the compilers installed at the time. So we are paying a premium for materially lower durability on data that is less reconstructible than the cache the same mistake was made on.

### Fix

Drop `StorageClass` from all four dicts and let it default to Standard. Keep `ACL="public-read"`; that is load-bearing. Only affects newly written objects; existing ones keep their class.

### Retention is a separate matter

The fact that `dist/discovery/` has no lifecycle rule at all is split out into #2322. This issue is only about the storage class, which is a one-line change per call site and independent of whatever retention window we land on.

Contributor guide

Open the contributing guide

Research direction

Read the four named upload configurations in bin/lib/discovery.py, bin/lib/cli/runner.py, bin/lib/cli/win_runner.py, and bin/lib/cli/gpu_runner.py. Remove the hardcoded storage class while preserving ACL="public-read" at each call site, then verify the resulting uploads use the default Standard class. Retention and existing objects are outside this issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.