compiler-explorer / compiler-explorer/infra
dist/discovery/ has no lifecycle rule and has grown to 163 GiB since 2021
- Dominant language
- Python
- Stars
- 434
- Forks
- 429
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 38
Description
> Written by Claude (an LLM), at Matt's request. Split out of #2321, which covers the storage class on the same objects. Same shape as #2290, but in the `compiler-explorer` bucket.
`dist/discovery/` has no expiration rule and has been accumulating since **2021-10-16**. It is now **163 GiB across 1,493 objects**.
The bucket's lifecycle config (`terraform/s3.tf:38`) has only two rules: a 32-day expiry on `cloudfront-logs`, and a bucket-wide `noncurrent_version_expiration` of 7 days. Nothing expires current objects under any other prefix.
### Where it is
| prefix | objects | size |
| --- | --- | --- |
| `dist/discovery/staging/` | 850 | 93.8 GiB |
| `dist/discovery/release/` | 587 | 67.0 GiB |
| `dist/discovery/beta/` | 35 | 2.3 GiB |
| `gpu` / `winstaging` / `winprod` | 21 | <0.1 GiB |
Individual files have grown from ~30 MB in 2021 to ~74 MB today as the compiler set has grown, so the accrual rate rises with every release.
**568 versions exist in both `staging/` and `release/` at byte-identical size, about 65 GiB of pure duplication.** That is expected rather than a bug: `copy_discovery` promotes the staging file to prod on deploy (`bin/lib/discovery.py`), so both copies are the same content. It just means the duplicate is only worth keeping for as long as the original is.
### Proposal: expire at 365 days
Age distribution as of 2026-08-22:
| age | objects | size |
| --- | --- | --- |
| < 90d | 87 | 4.6 GiB |
| 90d - 1y | 180 | 10.5 GiB |
| 1 - 3y | 610 | 95.7 GiB |
| > 3y | 616 | 52.3 GiB |
**91% of the bytes (148 GiB) are more than a year old.** We deploy far more often than annually, so a discovery file that has gone a year unused is not one we are going to roll back to.
One `expiration { days = 365 }` rule filtered on prefix `dist/discovery/` covers every environment, matching the existing single-prefix filter style. There are 32 `release/` objects from the last 90 days, so recent rollback targets are unaffected.
### Care needed
We must not delete a discovery that is still in use. An environment cannot come up without one, so removing the file for a live or still-deployable release breaks it. A year is well clear of that given how often we deploy.
### Money
Small, and worth saying so. At Standard pricing the whole prefix is about $3.75/month, so expiring 91% of it saves roughly $3.40/month. This is filed for the same reason as #2290: unbounded growth eventually stops being tidy, not because it is urgent.
Terraform changes here need `terraform fmt` and `terraform validate`; Matt applies.
Contributor guide
Research direction
Start at terraform/s3.tf:38 and inspect the existing lifecycle rules and prefix-filter style. Add the proposed 365-day expiration for dist/discovery/, then run terraform fmt and terraform validate. Done means the configuration covers all dist/discovery/ objects without changing the existing rules; Matt applies the Terraform change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, terraform
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100