Remote gRPC MIP solve: trailing log lines dropped by StreamLogs, causing test_cli_mip_remote to fail with objective=nan
@tmckayus is already working on this.
Since Jul 2, 2026.
- Dominant language
- Cuda
- Stars
- 1k
- Forks
- 233
- Avg merge
- 4d 4h
- Merged PRs (30d)
- 95
Description
Description
TestCuoptCliCPUOnly::test_cli_mip_remote in python/cuopt/cuopt/tests/linear_programming/test_cpu_only_execution.py failed on CI:
CI run: https://github.com/NVIDIA/cuopt/actions/runs/28603627708/job/84825516658?pr=1503
Job:conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4, latest-driver, latest-deps
AssertionError: Objective nan differs from expected 2.0 (rel error nan)
assert nan < 0.01
tests/linear_programming/test_cpu_only_execution.py:642: AssertionError
The failure occurred on PR #1503, which only touches documentation/Doxygen/Sphinx branding (cpp/doxygen/Doxyfile, cpp/include/cuopt/mathematical_optimization/cuopt_c.h comment, docs/cuopt/source/conf.py, and a logo image) — nothing related to the CLI, gRPC server, or MIP solver. So this is a pre-existing bug/flaky test unrelated to that PR's changes, surfaced incidentally.
Root cause
cuopt_cli run in remote-execution mode (CUOPT_REMOTE_HOST/CUOPT_REMOTE_PORT set) forwards the MIP solve to cuopt_grpc_server over gRPC. The client has no independent "objective summary" print for MIP — it depends entirely on log-line relay from the server via StreamLogs:
- Server side:
solve_mip_helper()(cpp/src/mip_heuristics/solve.cu:753) callssol.log_detailed_summary(), which emits the"Best objective {}, best bound {}, gap {}%."line (cpp/src/mip_heuristics/solver_solution.cu:239-250) — this is the line the CLI/test depends on to report the objective. Immediately afterward (solve.cu:755-756), ifsettings.sol_file != ""(set bycuopt_cli), a second line"Writing solution to file %s"is logged, right before the job is markedCOMPLETED. StreamLogs(cpp/src/grpc/server/grpc_service_impl.cpp:822-847) tails the per-job log file. Once it detects the job status has flipped toCOMPLETED/FAILED/CANCELLED, it performs exactly one extragetline()call ("read once more before closing the stream") to catch a line written after its last EOF poll, then sends the completion sentinel and returns.- If two or more lines are flushed to the log file in the window between the tailer's last EOF poll and the completion-status check (which is the normal case for MIP:
"Best objective..."followed by"Writing solution to file..."), only the first of those lines is guaranteed to be picked up — any additional line(s) can be silently dropped. This matches the observed symptom exactly:statusin_parse_cli_outputwas correctly parsed as"Optimal", but the"Best objective ..."line (or whichever line lost the race) never reached the client, leavingobjective_valueat itsnandefault.
This is a genuine race condition in the log-relay path (grpc_service_impl.cpp's StreamLogs), not a test-authoring bug. It is inherently flaky — timing-dependent on how many lines the worker flushes vs. how quickly the tailer polls job status after reaching EOF.
Suggested fix
In StreamLogs, after detecting a terminal job status, loop reading all remaining lines until EOF (rather than a single getline()) before sending the completion sentinel, e.g.:
while (std::getline(in, line)) {
...
if (!write_log_message(line, next_byte_offset, false)) { break; }
}
Reproduction
Not easily reproducible locally without a GPU + cuopt_grpc_server binary in a CPU-only remote-execution test environment; the race window is narrow, which is consistent with this test passing on most CI runs and failing intermittently. It failed on this specific run of PR #1503's CI, on the conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4, latest-driver, latest-deps job.
Environment
- CI job: https://github.com/NVIDIA/cuopt/actions/runs/28603627708/job/84825516658?pr=1503
- Test:
python/cuopt/cuopt/tests/linear_programming/test_cpu_only_execution.py::TestCuoptCliCPUOnly::test_cli_mip_remote
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.