Flaky: test_mip_incumbent_stream hangs in PROCESSING after the preceding cancel test kills the gRPC worker

Summary

TestGrpcClient::test_mip_incumbent_stream flakes in PR CI: the submitted MIP job never leaves PROCESSING, the client's 120 s poll expires, and the test fails with a misleading assertion. It passed on re-run of the same commit, so the failure is not related to the PR's changes.

Observed on run 31686594110, attempt 1 (conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4), PR #1700:

    terminal = _poll_until_complete(client, job_id, _MIP_NAMES)
>   assert terminal == JobStatus.COMPLETED
E   assert <JobStatus.PROCESSING: 1> == <JobStatus.COMPLETED: 2>
tests/linear_programming/test_grpc_client.py:276

Everything else in the job passed (1 failed, 126 passed, 9 skipped). Attempt 2 of the same run was fully green.

What the failure actually means

The job reached PROCESSING, which in check_job_status() (cpp/src/grpc/server/grpc_job_management.cpp:191) only happens once a worker has CAS-claimed the queue slot. So a worker took the job and then produced no result for 120 s — even though the test sets time_limit = 30. The solver time limit only bounds solve_mip(); it does not bound anything before the solve starts (worker CUDA/RMM init, problem transfer over the worker pipe), and nothing on the server side ever times a claimed job out. A wedged or very slow worker leaves the job in PROCESSING indefinitely.

Two properties of the test environment make that window much more likely on this runner:

1. The immediately preceding test SIGKILLs the worker. test_cancel_job (line 221) cancels a mid-solve swath1.mps job; cancel_job() sends SIGKILL to the worker PID (grpc_job_management.cpp:283). The worker-monitor thread then forks a replacement (grpc_server_threads.cpp:12-80), which must redo cudaSetDevice + allocate a fresh RMM pool (grpc_worker.cpp:43-83) while the killed process's GPU memory is still being reclaimed by the driver. test_mip_incumbent_stream submits into exactly that window.
2. Four xdist workers share one GPU. PR CI runs pytest -n 4 --dist loadgroup (ci/run_cuopt_pytests.sh:41), so three other GPU test processes are competing with the respawning worker for the L4.

Also worth noting: if init_worker_rmm_pool() throws (rmm::bad_alloc under contention), nothing catches it — init_worker_cuda_environment() and worker_process() have no try/catch, so the worker aborts and the monitor respawns it in a 100 ms loop with no backoff.

None of this is diagnosable from CI today, because the fixture starts the server with stdout=DEVNULL, stderr=DEVNULL (python/cuopt/cuopt/tests/fixtures/grpc_server_fixtures.py, spawn_server). We have no server or worker log for the failing attempt.

Finally, PR CI does not rerun: --reruns 2 --reruns-delay 5 is applied only when RAPIDS_BUILD_TYPE=nightly (ci/run_cuopt_pytests.sh:36-41). So one flake blocks the PR and the classifier reports it as "1 genuine test failure".

Proposed fixes

Observability (do first — otherwise the next occurrence is equally opaque)

• spawn_server() should tee server stdout/stderr to a file under the test's tmp dir instead of DEVNULL, and the grpc_server fixture should dump the tail of that file when a test in the class fails.
• Make _poll_until_complete raise an explicit timeout error (elapsed time + last observed status + incumbent count) rather than returning a non-terminal status that surfaces as PROCESSING != COMPLETED.

Test-level flake removal

• Don't let a worker-killing test run immediately before a solve test on the same server: either move test_cancel_job to the end of TestGrpcClient, or give it its own server instance (its own grpc_port_offset).
• After a cancel, wait for worker readiness before the next submit — needs a server-side health signal (below), or, as a stopgap, a fixture-level wait that submits a trivial probe job and waits for it to complete.
• Reduce poll pressure in _poll_until_complete: it issues status + result every 50 ms (~40 RPC/s), all taking tracker_mutex, which the result-retrieval thread also needs to mark the job COMPLETED. 200-250 ms is plenty.
• Consider --reruns 2 for PR CI too, at least for the grpc_server xdist group.

Server-side (the real fix)

• Add a job watchdog: if a claimed job exceeds its requested time_limit plus a margin, transition it to FAILED with a clear message instead of leaving it PROCESSING forever. Today a wedged worker is invisible to the client.
• Expose worker readiness (active_workers in SharedMemoryControl is already tracked) via the health/status RPC so clients can distinguish "queued behind a restart" from "running".
• Wrap init_worker_cuda_environment() / init_worker_rmm_pool() in a try/catch that logs and exits cleanly, and add backoff to the monitor's respawn path so a persistently failing worker doesn't fork-storm.