compiler-explorer / compiler-explorer/infra
wincache/ has no lifecycle rule and has grown to 1.26 TB / 12M objects
- Dominant language
- Python
- Stars
- 434
- Forks
- 429
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 38
Description
> Written by Claude (an LLM), at Matt's request, while auditing our S3 usage. Numbers are measured from the live bucket; the large prefixes were sampled by key shard rather than walked, so treat them as +/- 1%. Matt has spot-checked the main claims.
### The problem
`storage.godbolt.org` has exactly one lifecycle rule, and it covers `cache/`:
| ID | Prefix | Expiry |
| --- | --- | --- |
| Remove cached items | `cache/` | 1 day |
`wincache/` has no rule, so nothing has ever been deleted from it. It is now **12.2M objects and 1.26 TB**, with objects dating back to early 2023. That's 89% of the bucket's bytes and 75% of its objects.
### It looks like an oversight rather than a decision
The Linux and Windows caches are the same thing with a different prefix:
```
compiler-explorer.amazon.properties:11 cacheConfig=InMemory(25);S3(storage.godbolt.org,cache,us-east-1)
compiler-explorer.amazonwin.properties:8 cacheConfig=InMemory(50);S3(storage.godbolt.org,wincache,us-east-1)
```
Same key scheme (sha256 hex, with `_exec` companions), same storage class. The `cache/` rule dates from 2018 (`fefeda4f`); `wincache` was introduced in the main repo in 2023 (`7d192ccde`, #4761), and the oldest objects match that. Nothing revisited the lifecycle config when Windows was added.
Note an S3 lifecycle filter takes a single prefix, so this needs a second rule rather than a widened one.
The open question is the window. 1 day matches Linux, but Windows compiles are slower so a miss costs more. Nobody has hit-rate-against-age data, and 42 months is clearly wrong even if 1 day isn't obviously right. Also worth knowing: the first sweep would delete ~12M objects and Windows compile latency will suffer until the cache refills.
### Smaller things in the same bucket
- `compiler-info-cache/` has about 4 dead versions per live object — 458k noncurrent versions, 16.2 GB, going back 53 months. Versioning is on bucket-wide but `noncurrent_version_expiration` is scoped to `cache/`, so nothing reaps them. Plus ~307k delete markers from two mass-delete events in 2023-11 and 2025-08.
- Bucket versioning is enabled but not declared in terraform. We added `aws_s3_bucket_versioning` for the `compiler-explorer` bucket in `207a16f2` (2023-02-15) and never for this one; its versioning predates that.
- These caches are all on Reduced Redundancy, a legacy class that is now *more* expensive per GB than Standard. Nothing here needs it.
The money is small — the whole bucket is about $35/month, and only bounding `wincache/` moves that. This is filed for tidiness and because unbounded growth eventually stops being tidy, not because it's urgent.
Terraform changes here need `terraform fmt` and `terraform validate`; Matt applies.
Contributor guide
Research direction
Locate the Terraform definition for the storage.godbolt.org S3 bucket and inspect the existing cache/ lifecycle rule and versioning settings. Resolve the retention window for wincache/ with the maintainers, add the separate lifecycle configuration, then run terraform fmt and terraform validate; done means the rule is represented in Terraform and ready for Matt to apply.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, terraform
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100