compiler-explorer / compiler-explorer/ce-ci

Runner over-provisioning: multiple instances launched for a single slow-booting job

Open
#21 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
2
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Symptom

A single queued job can cause the scale-up Lambda to launch **several** runner instances instead of one. Observed on the Windows builder: **3 × `r5dn.xlarge` spot** instances launched for **one** `win-lib-build` job (2026-07-06).

This is **pre-existing** — not caused by the instance-type/gp3 changes committed the same day; those were verified green independently.

## Evidence (scale-up Lambda logs, `ce-ci-windows-x64-win-builder-scale-up`)

| Time | currentRunners | Job state | Action |
|---|---|---|---|
| 21:49:16 | 0 | queued | launch instance #1 |
| 21:56:42 | 1 | queued (still booting) | launch instance #2 |
| 21:57:08 | 2 | queued (still booting) | launch instance #3 |
| 22:01:26 | 3 | **not queued** | "No runner will be created, job is not queued" ✅ |

The job was finally picked up ~21:56 (Windows boot + register ≈ 7 min). Three separate `workflow_job:queued` scale-up requests (distinct SQS messageIds) each passed the "is the job still queued?" check *because the first runner was still booting*, so each launched another runner. The check only suppresses launches **after** the job is picked up — it can't during a long boot window.

## Why the extras linger (cost)

Scale-down runs every minute but respects two grace windows before terminating:
- **Boot grace** — Windows config sets `runner_boot_time_in_minutes: 20` (module default 5).
- **Idle grace** — `minimum_running_time_in_minutes` (module default).

So the two extra instances lived ~10 min each before being reaped. Per-incident cost is small on spot, but:
- It is **3× the intended capacity** per build.
- The same pattern on the **on-demand 32–64 vCPU Linux builder / x64 tiers** (`c6a.8xlarge`–`c7a.16xlarge`) would be **much** more expensive, and those are on-demand, not spot.

## Contributing config

- All our runner configs set **`delay_webhook_event: 0`** — the module default is **30**. That debounce window (event held invisible on the queue before the scale-up Lambda sees it) exists partly to let a booting runner register before duplicate/subsequent events are processed. We disabled it.
- Windows `runner_boot_time_in_minutes: 20` widens the window in which repeated queued events each trigger a launch.

## Mitigations to investigate

1. **Restore `delay_webhook_event` toward the module default (~30s)** — cheap, helps most on the faster-booting Linux tiers (boot ≈ 2 min); insufficient alone to bridge the ~7 min Windows boot.
2. **`enable_ephemeral_runners: true` (+ JIT)** — moves scale-up toward 1:1 job:runner and is the recommended posture; also the security improvement already flagged separately (runners currently persist state between jobs). Bigger change — needs its own testing.
3. Confirm whether newer module versions expose a "don't launch if a runner is already booting for this queued job" option, and whether the 3 duplicate `workflow_job:queued` deliveries originate from GitHub or from our webhook/EventBridge path.

## Acceptance

One queued job should result in exactly one runner under normal conditions, even with slow (Windows) boot times.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the scale-up Lambda and runner configuration, focusing on delay_webhook_event, runner_boot_time_in_minutes, and the webhook/EventBridge/SQS path described in the evidence. Trace duplicate queued-job handling and test the behavior with a slow-booting Windows runner; done means one queued job launches exactly one runner under normal conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions
Domain
ci-cd, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.