NVIDIA / NVIDIA/cuopt

[BUG] Flaky test: c_api/TimeLimitTestFixture.time_limit/2 (MIP) overshoots time budget on constrained CI runners

Open
#1,135 7 comments 0 reactions 1 assignee View on GitHub

@Kh4ster is already working on this.

Since Apr 22, 2026.

awaiting response
Dominant language
Cuda
Stars
1k
Forks
233
Avg merge
4d 4h
Merged PRs (30d)
95

Description

Summary

c_api/TimeLimitTestFixture.time_limit/2 (the MIP parametric case using /mip/supportcase22.mps, time_limit=15s, CUOPT_METHOD_DUAL_SIMPLEX) fails intermittently on CI when the reported solve_time exceeds target_solve_time + excess_allowed_time (15s + 3s = 18s).

Failing assertion

cpp/tests/linear_programming/c_api_tests/c_api_tests.cpp:64

EXPECT_NEAR(solve_time, target_solve_time, excess_allowed_time);  // excess = 3.0

Observed CI failure

  • solve_time = 18.398s, target 15s, tolerance 3.0s → over by ~0.4s.
  • Solver itself respected the limit: log shows Explored 0 nodes in 15.01s. and termination status Time limit (5).
  • CI environment per log: AMD EPYC 9554, only 12/12 threads visible (cgroup-limited on a 64-core host), 43 GiB RAM, NVIDIA H100 NVL, CUDA 13.1.

Root cause

Test reads cuOptGetSolveTime, which for MIP returns mip_solution_t::get_total_solve_time()stats_.total_solve_time. That field is recorded at cpp/src/mip_heuristics/solver.cu:489 (and similar check_time_limit guards) using a timer_t that was started in cpp/src/mip_heuristics/solve.cu:306before Papilo presolve.

Therefore solve_time includes:

  • Papilo presolve (OpenMP, ~2.59s in this run)
  • cuOpt presolve (~1.59s)
  • run_mip (LP root relaxation, primal heuristics, B&B)
  • Post-B&B serial wind-down inside mip_solver_t::solve(): branch_and_bound_status_future.get(), sol.compute_feasibility(), sol.test_variable_bounds(...) + device sync

It does NOT include Papilo postsolve, the full_sol rebuild, or C-API wrapping (those run after total_solve_time is finalized).

The 3.4s gap between B&B's internal stop (15.01s) and total_solve_time (18.40s) is the post-join serial wind-down. On CPU-thread-constrained runners:

  • Papilo presolve takes a larger share of the 15s budget (it's OpenMP-parallel — see cpp/src/mip_heuristics/solve.cu:412 passing settings.num_cpu_threads).
  • B&B uses omp_get_max_threads() - 1 worker threads (cpp/src/mip_heuristics/solver.cu:360).
  • The post-B&B compute_feasibility over the ~260K-row presolved problem becomes proportionally more expensive vs the budget.

Memory is not a factor at this problem size (~260K rows / 2.2M nz, peak well under 1 GiB).

Reproduction

  • CI: failure observed on H100 NVL CI runner with cgroup-limited 12-thread visibility.
  • Local (RTX 8000, compute 7.5): not reproducible — full c_api/TimeLimitTestFixture.time_limit/* set passes.

Suggested fixes (ranked)

  1. Measure solve_time from inside the time-limited region only so the field reflects what time_limit actually controls. Excludes presolve/post-solve serial work that the user's time_limit setting cannot bound.
  2. Per-case excess_allowed_time — keep 3s for LP cases, raise (e.g. 7s) for the MIP case to absorb host-side wind-down variance.
  3. Increase target_solve_time for the MIP case so post-solve overhead becomes a smaller % of the budget.

(1) is the principled fix; (2)/(3) are pragmatic unblocks.

Mitigation in progress

Disabling the /2 case in a follow-up PR while this is investigated; the LP /0 and /1 cases remain enabled.

Related

There is a stale TODO at cpp/src/dual_simplex/presolve.cpp:1517 mentioning this test name, but it is about a different (numerical assertion) failure mode, not this timing one.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.