NVIDIA / NVIDIA/cuopt

Remote gRPC MIP solve: trailing log lines dropped by StreamLogs, causing test_cli_mip_remote to fail with objective=nan

Open
#1,510 4 comments 0 reactions 1 assignee View on GitHub

@tmckayus is already working on this.

Since Jul 2, 2026.

awaiting response
Dominant language
Cuda
Stars
1k
Forks
233
Avg merge
4d 4h
Merged PRs (30d)
95

Description

Description

TestCuoptCliCPUOnly::test_cli_mip_remote in python/cuopt/cuopt/tests/linear_programming/test_cpu_only_execution.py failed on CI:

CI run: https://github.com/NVIDIA/cuopt/actions/runs/28603627708/job/84825516658?pr=1503
Job: conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4, latest-driver, latest-deps

AssertionError: Objective nan differs from expected 2.0 (rel error nan)
assert nan < 0.01
tests/linear_programming/test_cpu_only_execution.py:642: AssertionError

The failure occurred on PR #1503, which only touches documentation/Doxygen/Sphinx branding (cpp/doxygen/Doxyfile, cpp/include/cuopt/mathematical_optimization/cuopt_c.h comment, docs/cuopt/source/conf.py, and a logo image) — nothing related to the CLI, gRPC server, or MIP solver. So this is a pre-existing bug/flaky test unrelated to that PR's changes, surfaced incidentally.

Root cause

cuopt_cli run in remote-execution mode (CUOPT_REMOTE_HOST/CUOPT_REMOTE_PORT set) forwards the MIP solve to cuopt_grpc_server over gRPC. The client has no independent "objective summary" print for MIP — it depends entirely on log-line relay from the server via StreamLogs:

  • Server side: solve_mip_helper() (cpp/src/mip_heuristics/solve.cu:753) calls sol.log_detailed_summary(), which emits the "Best objective {}, best bound {}, gap {}%." line (cpp/src/mip_heuristics/solver_solution.cu:239-250) — this is the line the CLI/test depends on to report the objective. Immediately afterward (solve.cu:755-756), if settings.sol_file != "" (set by cuopt_cli), a second line "Writing solution to file %s" is logged, right before the job is marked COMPLETED.
  • StreamLogs (cpp/src/grpc/server/grpc_service_impl.cpp:822-847) tails the per-job log file. Once it detects the job status has flipped to COMPLETED/FAILED/CANCELLED, it performs exactly one extra getline() call ("read once more before closing the stream") to catch a line written after its last EOF poll, then sends the completion sentinel and returns.
  • If two or more lines are flushed to the log file in the window between the tailer's last EOF poll and the completion-status check (which is the normal case for MIP: "Best objective..." followed by "Writing solution to file..."), only the first of those lines is guaranteed to be picked up — any additional line(s) can be silently dropped. This matches the observed symptom exactly: status in _parse_cli_output was correctly parsed as "Optimal", but the "Best objective ..." line (or whichever line lost the race) never reached the client, leaving objective_value at its nan default.

This is a genuine race condition in the log-relay path (grpc_service_impl.cpp's StreamLogs), not a test-authoring bug. It is inherently flaky — timing-dependent on how many lines the worker flushes vs. how quickly the tailer polls job status after reaching EOF.

Suggested fix

In StreamLogs, after detecting a terminal job status, loop reading all remaining lines until EOF (rather than a single getline()) before sending the completion sentinel, e.g.:

while (std::getline(in, line)) {
  ...
  if (!write_log_message(line, next_byte_offset, false)) { break; }
}

Reproduction

Not easily reproducible locally without a GPU + cuopt_grpc_server binary in a CPU-only remote-execution test environment; the race window is narrow, which is consistent with this test passing on most CI runs and failing intermittently. It failed on this specific run of PR #1503's CI, on the conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4, latest-driver, latest-deps job.

Environment

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.