compiler-explorer / compiler-explorer/infra

dist/discovery/ has no lifecycle rule and has grown to 163 GiB since 2021

Open Beginner friendly
#2,322 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
434
Forks
429
Avg merge
4d 21h
Merged PRs (30d)
38

Description

> Written by Claude (an LLM), at Matt's request. Split out of #2321, which covers the storage class on the same objects. Same shape as #2290, but in the `compiler-explorer` bucket.

`dist/discovery/` has no expiration rule and has been accumulating since **2021-10-16**. It is now **163 GiB across 1,493 objects**.

The bucket's lifecycle config (`terraform/s3.tf:38`) has only two rules: a 32-day expiry on `cloudfront-logs`, and a bucket-wide `noncurrent_version_expiration` of 7 days. Nothing expires current objects under any other prefix.

### Where it is

| prefix | objects | size |
| --- | --- | --- |
| `dist/discovery/staging/` | 850 | 93.8 GiB |
| `dist/discovery/release/` | 587 | 67.0 GiB |
| `dist/discovery/beta/` | 35 | 2.3 GiB |
| `gpu` / `winstaging` / `winprod` | 21 | <0.1 GiB |

Individual files have grown from ~30 MB in 2021 to ~74 MB today as the compiler set has grown, so the accrual rate rises with every release.

**568 versions exist in both `staging/` and `release/` at byte-identical size, about 65 GiB of pure duplication.** That is expected rather than a bug: `copy_discovery` promotes the staging file to prod on deploy (`bin/lib/discovery.py`), so both copies are the same content. It just means the duplicate is only worth keeping for as long as the original is.

### Proposal: expire at 365 days

Age distribution as of 2026-08-22:

| age | objects | size |
| --- | --- | --- |
| < 90d | 87 | 4.6 GiB |
| 90d - 1y | 180 | 10.5 GiB |
| 1 - 3y | 610 | 95.7 GiB |
| > 3y | 616 | 52.3 GiB |

**91% of the bytes (148 GiB) are more than a year old.** We deploy far more often than annually, so a discovery file that has gone a year unused is not one we are going to roll back to.

One `expiration { days = 365 }` rule filtered on prefix `dist/discovery/` covers every environment, matching the existing single-prefix filter style. There are 32 `release/` objects from the last 90 days, so recent rollback targets are unaffected.

### Care needed

We must not delete a discovery that is still in use. An environment cannot come up without one, so removing the file for a live or still-deployable release breaks it. A year is well clear of that given how often we deploy.

### Money

Small, and worth saying so. At Standard pricing the whole prefix is about $3.75/month, so expiring 91% of it saves roughly $3.40/month. This is filed for the same reason as #2290: unbounded growth eventually stops being tidy, not because it is urgent.

Terraform changes here need `terraform fmt` and `terraform validate`; Matt applies.

Contributor guide

Open the contributing guide

Research direction

Start at terraform/s3.tf:38 and inspect the existing lifecycle rules and prefix-filter style. Add the proposed 365-day expiration for dist/discovery/, then run terraform fmt and terraform validate. Done means the configuration covers all dist/discovery/ objects without changing the existing rules; Matt applies the Terraform change.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, terraform
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.