compiler-explorer / compiler-explorer/ce-ci
Evaluate ephemeral runners (one job per instance) to remove workspace-reuse cleanup hacks
- Dominant language
- HCL
- Stars
- 2
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Motivation
Runners are currently **non-ephemeral** (`enable_ephemeral_runners: false`), so an instance can be **reused** across jobs. Every CI/interactive workflow in `infra` compensates with a manual **"Clean workspace"** step to scrub leftover state between runs:
```yaml
- name: Clean workspace
run: find "$GITHUB_WORKSPACE" -mindepth 1 -delete # some variants use sudo
```
Present in (at least): `install.yml`, `consolidate.yml`, `bespoke-build.yaml`, `bespoke-build-arm64.yaml`, `adhoc-command.yml`, `adhoc-command-lin-builder.yml`, `cefs-gc.yml`.
**Ephemeral runners** (`enable_ephemeral_runners: true`) make each instance run **exactly one job and then self-terminate**, so the workspace is always pristine and these cleanup steps become unnecessary. It's also the module's recommended security posture (README: "Runners are created on-demand and terminated after use").
## What changes (config-only — no AMI rebuild)
Confirmed the baked AMIs already support ephemeral + JIT on **both** platforms:
- Linux `packer/start-runner.sh`: ephemeral path runs `./run.sh --jitconfig` and, on exit, `aws ec2 terminate-instances` (self-terminate).
- Windows `packer/start-runner.ps1`: same via `run.cmd --jitconfig`.
So enabling is per-tier `enable_ephemeral_runners: true` in `templates/runner-configs/*.yaml`. Requirements met:
- Ephemeral requires `workflow_job` webhook events (scale-up errors otherwise) — we already use these.
- JIT config is auto-enabled for ephemeral (`enable_jit_config` default follows ephemeral).
## Benefits
- **Removes the "Clean workspace" hacks** — pristine workspace guaranteed per job.
- **Cleans *all* per-run state**, not just `$GITHUB_WORKSPACE` (e.g. builders leave `/tmp/build`, `/tmp/staging`, `~/.conan` on reused instances today).
- **Security**: no cross-job state on instances that hold CE IAM roles / secrets.
## Trade-offs (be honest)
- **Cold boot per job.** Non-ephemeral runners can be reused by a following job; ephemeral pays boot every time. Negligible for **builders** (long jobs amortise a 2–7 min boot) but noticeable on the frequent, short **CI/small** tier — worst on **Windows** (~7 min boot; `runner_boot_time_in_minutes: 20`).
- Mitigation: `idle_config` (keep `idleCount` warm runners on a `cron`/`timeZone` schedule) or `pool_config` (pre-created pool). Both cost idle-instance money; tune to business hours.
- **Cold local caches.** Fresh instance ⇒ cold `~/.conan` download cache on builders (re-downloads base deps from `ceserver` — network, not rebuild). **Low impact**: `start-builder.sh` already runs `rm -rf infra && git clone` + `make ce` fresh each run and sources compilers via cefs, so it's already close to stateless. Compiler/library caches are server/cefs-backed, not per-runner-local.
## Interaction with over-provisioning (#21) — orthogonal, verify in pilot
Ephemeral does **not** automatically fix #21, and could interact with it:
- `enable_job_queued_check` **defaults to disabled for ephemeral** (enabled for non-ephemeral). It's the guard that (partially) limits duplicate scale-ups.
- JIT config is **label-scoped, not job-scoped** (`createJitConfig` registers a runner for the tier's labels, not a specific job id) — so a duplicate `workflow_job:queued` burst could still launch multiple JIT runners; the extras find no job and are reaped.
- There's an experimental **`job_retry`** feature for ephemeral (requeues if a job isn't picked up) worth understanding.
- **Action:** in the pilot, measure scale-up behaviour and consider explicitly setting `enable_job_queued_check: true` even for ephemeral.
## Proposed rollout (cautious, per-tier bool, reversible)
1. **Pilot on the builders** (`lin-builder` + `win-builder`) — highest benefit (clean state matters most), lowest overhead (long jobs, infrequent), already near-stateless.
2. Test: run `lin-lib-build` (fmt/g151) twice back-to-back; confirm each gets a **fresh instance + empty workspace/tmp**, build isn't materially slower from cold conan cache, and scale-up stays ~1:1.
3. If good, extend to CI tiers (`x64`/`small`/`medium`/`arm64`), adding a small `idle_config` warm pool only if cold-boot latency on `small` proves annoying.
4. **Coordinated infra PR** to strip the now-redundant "Clean workspace" steps (do this last, once ephemeral is proven — the steps are harmless to keep during transition).
## Open questions to resolve during pilot
- Actual cold-boot penalty per tier, and whether a warm pool is warranted for `small`/Windows.
- Cold `~/.conan` cache impact on builder wall-clock (measure fmt build ephemeral vs current).
- Ephemeral scale-up behaviour vs #21 — does it need `enable_job_queued_check: true` and/or `job_retry`?
## Notes
- Config-only, no AMI rebuild (verified above).
- Related: over-provisioning #21; ephemeral is also the security hardening flagged during the 2026-07-06 review.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with templates/runner-configs/*.yaml and read the module README, then inspect packer/start-runner.sh and packer/start-runner.ps1 to verify the existing ephemeral paths. Pilot lin-builder and win-builder by running lin-lib-build twice, measuring boot and cache impact, and checking scale-up behavior against #21; done means fresh instances, acceptable build time, and roughly 1:1 runner creation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, github-actions, yaml
- Domain
- ci-cd, devops, infrastructure, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100