test_cpu_x86_llamacpp_floor's contended-leg case is INTERMITTENT on an unchanged tree: one exit-4 NO_QUIET_WINDOW against the asserted exit-2 GIVING_UP, cause not established

Row: -

tests/scripts/test_cpu_x86_llamacpp_floor.py::test_a_contended_leg_is_discarded_and_never_summarised fails when the host is genuinely busy, and passes when it is not. That is a defect in the suite whose whole purpose is to make contention handling deterministic.

Observed

During a preflight on a box at load average: 36.37, 35.36, 34.79 (four sessions running agent-preflight.sh concurrently):

AssertionError: 4 != 2 : ours rep=1 START load=32.35 35.66 34.85 builders=0
ours rep=1 END exit=0 load=32.35 35.66 34.85 builders=0 foreign=0%
ours rep=1 DISCARDED (exit=0 builders_after=0 foreign=0%)
pair rep=1 RETRY
ours rep=1 START load=32.35 35.66 34.85 builders=0

Rerun alone on the same tree minutes later: 1 passed in 93.29s.

The two exits are different stop conditions, so this is not a near-miss

• exit 2 — scripts/cpu-x86-llamacpp-floor.sh:336, GIVING_UP too many discards, after 24 discarded attempts. This is what the test asserts.
• exit 4 — :127, NO_QUIET_WINDOW after ${waited}s, from wait_quiet. The box never became quiet enough to start a leg.

The test forces every leg to be discarded with FOREIGN_MAX="-1" and expects the harness to reach the 24-discard ceiling. On a loaded host the quiet gate fires first, so the harness stops for a different and legitimate reason — and the assertion reads that as a failure.

Attribution — it is not the change under test

Established before blaming the host, because "this commit touched the area" is not "this commit caused the failure":

• tests/scripts/test_cpu_x86_llamacpp_floor.py is byte-identical to origin/main (git diff --stat origin/main HEAD -- <path> is empty).
• scripts/cpu-x86-llamacpp-floor.sh, the harness it actually executes, is byte-identical to origin/main too.
• The branch it failed on merges three PRs that touch neither: two are records/evidence only, the third is the GLM router dtype.

Mechanism — a hypothesis, NOT established

I did not observe the NO_QUIET_WINDOW line's busy= value, so treat this as the leading candidate rather than a finding:

busy_pct() {
  b0=$(stat_busy); t0=$(stat_total)
  sleep "$BUSY_WINDOW"
  b1=$(stat_busy); t1=$(stat_total)
  db=$((b1 - b0)); dt=$((t1 - t0))
  ...
  echo $((100 * db / dt))
}

stat_busy and stat_total are two separate reads of /proc/stat, so db and dt sample slightly different intervals. The test runs with BUSY_WINDOW=0, which makes that skew the entire measurement rather than a rounding error on a real window. Under load the gap between the two reads widens, and db > dt becomes possible — which would put p above 100, above the test's QUIET_BUSY=100 ceiling, so the gate never opens and WAIT_TIMEOUT=30 is reached.

Ruled out already: the builders gate. builders() uses pgrep -x (exact process name), and the test passes BUILDERS="no-such-process-name", so it is 0 — and the captured output confirms builders=0 on every line.

To confirm or kill the hypothesis, capture the busy= value from the NO_QUIET_WINDOW line under load. If p > 100, reading t0/t1 from the same /proc/stat snapshot as b0/b1 fixes it.

Why this is worth fixing rather than retrying

A test that flakes with host load is a test whose green means "the box was quiet", not "the code is right" — and this one gates the contention logic itself. It also makes every concurrent preflight on this machine slightly likelier to red for a reason no commit caused, which is the same class of problem as #2950.

Start with tests/scripts/test_cpu_x86_llamacpp_floor.py::test_a_contended_leg_is_discarded_and_never_summarised and the wait_quiet/busy_pct logic in scripts/cpu-x86-llamacpp-floor.sh. Run the test under host load and capture the NO_QUIET_WINDOW busy value, then compare it with the exit-2 discard path. Done means the contention test remains deterministic under load while preserving its intended discard assertion.