openai / openai/codex-plugin-cc

test: broker-spawn integration tests flake under host load — widen/tune the waitFor budget and reap leaked processes

Open
#632 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
33.3k
Forks
2.3k
PR merge metrics
No merged PRs in 30d

Description

Update (2026-08-10): the primary failure mode originally described here turned out to be a product race, not test-infrastructure flakiness — #623's enqueue path rewrote the job file (truncate-in-place) at the exact moment the detached worker's startup read it, killing the worker silently and leaving the job queued forever; host load only modulated the collision probability. That race is fixed in #623 (writeJobFile now writes a temp file and renames it into place, with a torn-record regression test), and the suite passes 144/144 on the host that showed the failures. This issue's remaining scope is the two test-infrastructure hardenings below, which stand on their own.

Problem

The runtime tests that spawn the full background-task chain — detached worker → workspace broker → fake codex app-server → unix-socket handshake → turn/start — gate on waitFor(..., { timeoutMs: 15000 }). On a loaded host (a VM reserving most cores, an endpoint-security agent scanning every spawned process), that chain can exceed 15 s even with no functional defect, and the test times out. Observed on 2026-08-10 while validating #623: even after the product race above was fixed, one run had a test miss the budget by ~1.2 s purely from suite parallelism plus per-spawn scanning overhead.

A compounding factor: a test run that fails or is interrupted leaks its temp-workspace broker and app-server processes (codex-plugin-test-* cwd). These accumulated to ~150 idle node processes across a few failed runs, degrading subsequent runs until manually swept with pkill -f codex-plugin-test-. A near-green run (142/143) still left 66 such processes behind, so teardown is best-effort even on the passing path.

Proposal

  • Make the wait budget scalable: apply an env multiplier (e.g. CODEX_COMPANION_TEST_TIMEOUT_SCALE) inside the waitFor helper (tests/runtime.test.mjs:20) so loaded hosts and slow CI runners can widen every budget in one place, keeping the defaults for fast machines.
  • Reap spawned processes on test teardown unconditionally (including timeout and passing paths) so runs cannot leak workspace processes and poison later runs.

Out of scope

Product-code changes. The torn-read race originally blamed on load is fixed in #623; the spawn chain's inherent cost is by design for these integration-level tests.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tests/runtime.test.mjs at the waitFor helper around line 20, then trace how the detached worker, broker, and app-server processes are spawned and torn down. Add the configurable timeout scaling and ensure teardown runs on failures, interruption, and passing paths; done means the runtime suite tolerates slower hosts without leaked codex-plugin-test-* processes.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.