NVIDIA / NVIDIA/cuopt

CI: investigate conda-cpp-tests jobs hitting the 40m timeout

Open
#1,131 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

awaiting response
Dominant language
Cuda
Stars
1k
Forks
233
Avg merge
4d 4h
Merged PRs (30d)
95

Description

Context

The `conda-cpp-tests` job in the `pr` workflow has been hitting the outer 40-minute `timeout` wrapper around `./ci/run_ctests.sh` (in `ci/test_cpp.sh`) and failing CI as a result.

Example failing run (PR #1035):

As a short-term unblock, PR #1130 bumps the outer timeout from 40m to 50m. This issue tracks the underlying investigation — we should understand why the job is now running longer, not just raise the ceiling indefinitely.

Investigation items

  • Break down the ctest run-time per test binary (e.g. `ctest --output-on-failure -V --timeout N` or parse the `Test.xml`) to identify which suites have regressed.
  • Compare wall-clock of `conda-cpp-tests` on `main` over the last ~30 days — is this a recent regression or a gradual drift?
  • Check whether slow tests are concentrated in one area (MILP, LP, routing GES, grpc integration) so we can target the fix.
  • Confirm the regression is not specific to one runner pool or GPU (H100 vs. others in the matrix).
  • Decide: (a) reduce test runtime at the source, (b) split into parallel jobs, (c) gate slow tests behind a nightly job, or (d) accept a permanently higher ceiling.

Acceptance criteria

  • Root cause is documented on this issue.
  • Either the offending tests are sped up / skipped on PRs, or a CI topology change is proposed and merged, such that the PR `conda-cpp-tests` job consistently finishes well under the outer timeout.

Related

  • #1130 — bump the outer timeout from 40m to 50m as a stop-gap

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pr workflow and ci/test_cpp.sh, then inspect how ./ci/run_ctests.sh is wrapped by the 40-minute timeout. Use ctest output or Test.xml to compare per-test durations, review conda-cpp-tests runs on main over the last 30 days, and compare runner/GPU results. Done means the root cause is documented and the job consistently finishes well under the outer timeout through a merged runtime or CI-topology change.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, github-actions
Domain
ci-cd, performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.