The "large" runner label is not a constraint: large jobs run on small runners

> Written by Claude (an LLM), at Matt's request, after investigating a run of nightly OOM kills. Numbers below are measured from the GitHub API and kernel OOM reports; Matt has seen the analysis.

## The problem

`"large"` does not mean "a large machine". It means **no size constraint at all**, so large jobs routinely run on small runners.

GitHub matches `runs-on` as *"the runner must carry at least these labels"*. Fewer labels therefore makes a job schedulable on **more** machines, not fewer. Our pools register as:

| pool | registered labels |
|---|---|
| linux-x64 ("large") | `{self-hosted, ce, linux, x64}` |
| linux-x64-medium | `{self-hosted, ce, linux, x64, medium}` |
| linux-x64-small | `{self-hosted, ce, linux, x64, small}` |

A small runner's set is a **superset** of the large request, so it satisfies it. The same holds for scale-up routing: as #20 puts it, `exactMatch` means "job labels must be a subset of the runner's labels", so a bare `[self-hosted, ce, linux, x64]` job matches the small pool's matcher too.

The intent in `compiler-workflows/make_builds.py` was "untagged builds land on large, so we don't have to tag every build". The implementation says "untagged builds land anywhere".

## Evidence

`clang_llvmflang` job records, sampled 2026-04-30 to 2026-08-14 (107 records):

| runner pool | runs | successes |
|---|---|---|
| `ce-x64-small` | 31 | **0** |
| `ce-x64` / `ce-x64-medium` | 76 | 15 |

Thirty-one runs on a 16 GiB box, not one success, for four months. Each died as an OOM kill at ~03:00 and nothing surfaced it. `circt_trunk`'s only recent failure was the same thing: it is not memory-hungry, it simply landed on a 16 GiB machine.

compiler-explorer/clang-builder#119 has since made the job counts derive from the machine's RAM, so a misrouted build now runs slowly instead of dying. **This issue is therefore no longer about availability, it is about jobs silently getting a machine a quarter the intended size.**

## Who asks for the bare form today

- `compiler-workflows`: all ~57 generated `build-daily-*.yml` (size defaults to `large`)
- `infra/.github/workflows/bespoke-build.yaml:68` and `infra/.github/workflows/adhoc-command.yml:32` — both `inputs.size == 'large' && fromJSON('["self-hosted","ce","linux","x64"]')`

Everything else is explicit (`small`, `medium`, `arm64`, `lin-builder`, `win-builder`, `admin`) and is unaffected.

## The fix, and why it can be rolled out without breaking anything

Give the large pool a label of its own, **while keeping the bare matcher**, so existing callers keep working during the transition. `modules/multi-runner/main.tf` at our pinned v7.9.0 unions every matcher into the registered labels:

```hcl
runner_extra_labels = { for k, v in var.multi_runner_config :
k => sort(setunion(flatten(v.matcherConfig.labelMatchers), compact(v.runner_config.runner_extra_labels))) }
```

So with two matchers the pool registers `{self-hosted, ce, linux, x64, large}` and still matches bare-form jobs for both scheduling and scale-up.

### Rollout

- [x] **1. ce-ci** — #25 (merged and applied 2026-08-14) — `templates/runner-configs/linux-x64.yaml`, add a second matcher, keeping the existing one:
```yaml
labelMatchers:
- [self-hosted, ce, linux, x64, large]
- [self-hosted, ce, linux, x64]
```
`terraform fmt` + `validate`, then apply. **Non-breaking**: nothing requests `large` yet, and bare-form callers still match.
- [x] **2. Verify** a large runner comes up carrying the `large` label before touching any caller (e.g. run a `bespoke-build` and check the runner name/labels, or inspect the runner list).
- [x] **3. compiler-workflows** — compiler-explorer/compiler-workflows#78 (merged 2026-08-14) — `SIZE_TO_LABELS["large"]` becomes `[ 'self-hosted', 'ce', 'linux', 'x64', 'large' ]`, then `make build-yamls` to regenerate. The "untagged defaults to large" behaviour is preserved: `make_builds.py` still defaults `size="large"`, that default now just means something.
- [x] **4. Confirm** the daily builds are landing on `ce-x64_*` runners only (the job's `runner_name` is in the API), then
- [x] **5. infra** — compiler-explorer/infra#2301 (merged 2026-08-14) — `bespoke-build.yaml` and `adhoc-command.yml`. The `size == 'large'` special case is *deleted* rather than extended: once all three sizes have a label, the conditional collapses to the `format()` branch already used for small and medium.
- [ ] **6. ce-ci** — once nothing requests the bare form, drop the second matcher so `large` is the only way in. (No PR yet, deliberately: it is only safe after 3 and 5 have landed and been observed.)

Steps 1, 3 and 5 are each independently deployable and reversible. **Order matters**: doing 3 or 5 before 1 means large jobs request a label no runner has, and they queue forever.

## Relationship to #20

This is a step *towards* #20, not a divergence. Under `bidirectionalLabelMatch` the match becomes exact in both directions, so a bare `[self-hosted, ce, linux, x64]` job would match only the large pool — which incidentally fixes this problem too, but on the v8 timeline (#20 estimates late 2026) and only after the builder short forms are sorted.

Doing the above first means that when #20 lands, `compiler-workflows` and `infra` are already sending the exact label set the large pool registers, so that migration has one less thing to fix rather than one more.

Worth noting on #20 as well: it is currently written up purely as "these builder workflows will break". It is also a **fix** for misrouting that is happening today, which is a point in its favour that has not been costed.

---

## PRs

| step | PR | state |
|---|---|---|
| 1 | compiler-explorer/ce-ci#25 | ✅ merged and applied |
| 3 | compiler-explorer/compiler-workflows#78 | ✅ merged |
| 5 | compiler-explorer/infra#2301 | ✅ merged |
| 6 | not raised yet | after 3 and 5 are observed working |

Steps 3 and 5 are drafts on purpose: merging either before step 1 is applied makes those jobs request a label no runner carries, and they queue indefinitely rather than failing.

One thing the tooling caught that is worth recording: both `compiler-workflows` and `infra` keep a `.github/actionlint.yaml` allowlist of valid self-hosted labels, and the pre-commit hook rejected `large` until it was added. So a fourth place needed to know about the label, which nothing in the plan above had predicted.

---

### Step 2 result (2026-08-14)

Dispatched `adhoc-command.yml` from infra#2301's branch with `size=large`, so the job requested the new label without anything being merged:

```
labels = ["self-hosted","ce","linux","x64","large"]
runner = ce-x64_i-0a0b133b5809271fc <- large pool
status = success
```

The pool registers `large`, scale-up matches on it, and a freshly launched large runner took the job. Steps 3 and 5 are unblocked.

**Unrelated drift surfaced by the same apply, for the record:** the plan carried 25 changes, only two of which were this work (the `RUNNER_LABELS` env on `linux-x64` scale-up, and the `runner_matcher_config` SSM parameter). The other 23 were lambda `source_code_hash` updates: the deployed lambdas matched no v7.x release and none of v6.3.0-v6.10.1 either, i.e. they were years older than the `v7.9.0` pinned in both `main.tf` files. Cause is that `lambdas-download/` is a separate terraform root whose zips are gitignored, so bumping the version does not refresh them without a second, manual `terraform apply`. That has now been corrected as part of this apply, and the successful dispatch above confirms the upgraded webhook/dispatcher path works. A guard against it recurring is #26.

### Step 4 result (2026-08-14)

Dispatched `circt_trunk` and `clang_llvmflang` manually rather than waiting for the 01:00 schedule. Both were picked up within ~2 minutes by freshly launched **c6a.8xlarge** (32 vCPU / 64 GiB) instances from the large pool, and neither touched a small runner:

```
circt ce-x64_i-052df67d8b843de65 success, 45.3 min
llvmflang ce-x64_i-0ffad1bfa9d0516cc success, 114.9 min
```

`clang_llvmflang` had failed 60 consecutive nightly runs; this is its first success since 2026-06-16. It needed both halves: a guaranteed 64 GiB machine from this work, and RAM-derived compile-job counts from compiler-explorer/clang-builder#119 (8 jobs instead of the hardcoded 12). Wall clock went from 79 min at 12 jobs to 114.9 at 8, as expected.

Only step 6 remains, deliberately left for a few days of observation.

Start with modules/multi-runner/main.tf and templates/runner-configs/linux-x64.yaml, then inspect the remaining bare-form callers in compiler-workflows and infra. Observe several days of large-runner scheduling and confirm no jobs request the bare matcher; done means the second matcher can be removed without queuing or misrouting jobs.

Written by the indexing model from the issue text.