Merge queue: hive/hive-eest jobs wait a median 13m (p95 1h44m, max 4h10m) for a runner

## Summary

On the merge queue (`CI Gate`, `merge_group` event), `hive` and `hive-eest` jobs spend far longer **waiting for a `hive`-group runner** than they spend executing.

Over the 7 days ending **2026-06-02** (158 `CI Gate` merge-queue runs, 3,075 hive/eest jobs that actually ran):

- **Median time waiting for a runner: 13m19s** — *longer* than the median job run time of **7m36s**.
- **p95 wait: 1h44m. Worst observed wait: 4h10m.**
- **79%** of jobs waited >1 min; **69%** waited >5 min; **56%** waited >10 min.
- Cumulative queue wait was **~1,330 runner-hours vs ~624 hours of actual execution** — jobs spent roughly **2× longer waiting than running**.

This wait sits on the critical path for merging and directly delays merge-queue throughput.

## How this was measured

- **Scope:** all `CI Gate` workflow runs with `event=merge_group` created in the last 7 days (158 runs).
- **Jobs:** the matrix jobs of `test-hive.yml` (`hive / test-hive (...)`) and `test-hive-eest.yml` (`hive-eest / test-hive-eest (...)`), which target `runs-on: group: hive`.
- **Metric:** queue wait = `job.started_at − job.created_at` from the Actions jobs API. `created_at` is stamped when the job enters the **queued** state (after `needs` clear). `hive` depends only on the trivial `changes` job and `hive-eest` has no `needs`, so this gap is **pure runner-acquisition wait**, not dependency wait.
- **Population:** 3,634 hive/eest job instances total → **3,075 ran on a real runner** (clean wait, reported below); 559 were cancelled by merge-queue reshuffles before ever acquiring a runner (≈0 wait, excluded from wait stats).

## Queue wait by job class (jobs that ran on a runner)

| class | jobs | median | mean | p90 | p95 | max | >1m | >5m | >10m |
|---|---|---|---|---|---|---|---|---|---|
| `hive` | 1,587 | **17m10s** | 31m17s | 80m34s | 121m33s | 250m27s | 1,436 | 1,275 | 1,013 |
| `hive-eest` | 1,488 | **8m13s** | 20m27s | 54m00s | 84m49s | 211m56s | 1,001 | 852 | 716 |
| **ALL** | 3,075 | **13m19s** | 26m03s | 69m21s | 104m01s | **250m27s** | 2,437 | 2,127 | 1,729 |

## How often it happens (per merge-queue run, of 158)

| at least one hive/eest job waited > | # runs | % of runs |
|---|---|---|
| 1 min | 149 | 94% |
| 5 min | 147 | 93% |
| 10 min | 125 | 79% |
| 20 min | 93 | 59% |

(7 of the 158 runs had no hive job run at all — fully cancelled by reshuffles before any hive job started.)

## Worst individual waits

| wait | job | run |
|---|---|---|
| **4h10m** | `hive / test-hive (ethereum/rpc-compat, .*, serial)` | [26564641941](https://github.com/erigontech/erigon/actions/runs/26564641941) |
| 3h36m | `hive / test-hive (ethereum/rpc-compat, .*, serial)` | [26573583585](https://github.com/erigontech/erigon/actions/runs/26573583585) |
| 3h32m | `hive-eest / test-hive-eest (osaka, parallel)` | [26564795461](https://github.com/erigontech/erigon/actions/runs/26564795461) |
| 3h23m | `hive-eest / test-hive-eest (cancun, parallel)` | [26564795461](https://github.com/erigontech/erigon/actions/runs/26564795461) |

## Observed capacity vs demand

- The `hive` runner group is **16 machines, one job per machine = 16 concurrent slots** (`devops-bm-ghrunner-n2`…`n17`).
- Each `CI Gate` run dispatches **23 hive-group jobs** (12 `hive` + 11 `hive-eest`). Since 23 > 16, a backlog forms within every single run before any cross-run contention.
- The merge queue runs multiple speculative `CI Gate` runs concurrently, so demand stacks: the deepest observed backlog was **126 hive/eest jobs waiting at once**, with the pool pegged at **16/16 running**.
- Average slot utilization over the week was only **~23%** — the load is bursty (idle between merge-queue waves, fully saturated during them), which is why per-host CPU/memory/disk dashboards look healthy while jobs queue for minutes-to-hours.

## Reproduction

The figures above come from the Actions API: list `CI Gate` runs with `event=merge_group` for the last 7 days, then for each run read its jobs and compute `started_at − created_at` for jobs whose name matches `test-hive`/`test-hive-eest`.