CI: investigate conda-cpp-tests jobs hitting the 40m timeout
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 1k
- Forks
- 233
- Avg merge
- 4d 4h
- Merged PRs (30d)
- 95
Description
Context
The `conda-cpp-tests` job in the `pr` workflow has been hitting the outer 40-minute `timeout` wrapper around `./ci/run_ctests.sh` (in `ci/test_cpp.sh`) and failing CI as a result.
Example failing run (PR #1035):
- Run: https://github.com/NVIDIA/cuopt/actions/runs/24721054667/job/72360181090?pr=1035
- Matrix: CUDA 13.1.1, Python 3.14, amd64, Ubuntu 24.04, H100, latest driver, latest deps
- Duration: 2026-04-21T16:55:08Z → 2026-04-21T17:40:09Z (~45 min wall-clock, killed by the outer `timeout 40m`)
- Conclusion: failure
- Runner: `609e-l-amd-g-h100-l-1-9rdw2-runner-v49qn`
As a short-term unblock, PR #1130 bumps the outer timeout from 40m to 50m. This issue tracks the underlying investigation — we should understand why the job is now running longer, not just raise the ceiling indefinitely.
Investigation items
- Break down the ctest run-time per test binary (e.g. `ctest --output-on-failure -V --timeout N` or parse the `Test.xml`) to identify which suites have regressed.
- Compare wall-clock of `conda-cpp-tests` on `main` over the last ~30 days — is this a recent regression or a gradual drift?
- Check whether slow tests are concentrated in one area (MILP, LP, routing GES, grpc integration) so we can target the fix.
- Confirm the regression is not specific to one runner pool or GPU (H100 vs. others in the matrix).
- Decide: (a) reduce test runtime at the source, (b) split into parallel jobs, (c) gate slow tests behind a nightly job, or (d) accept a permanently higher ceiling.
Acceptance criteria
- Root cause is documented on this issue.
- Either the offending tests are sped up / skipped on PRs, or a CI topology change is proposed and merged, such that the PR `conda-cpp-tests` job consistently finishes well under the outer timeout.
Related
- #1130 — bump the outer timeout from 40m to 50m as a stop-gap
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the pr workflow and ci/test_cpp.sh, then inspect how ./ci/run_ctests.sh is wrapped by the 40-minute timeout. Use ctest output or Test.xml to compare per-test durations, review conda-cpp-tests runs on main over the last 30 days, and compare runner/GPU results. Done means the root cause is documented and the job consistently finishes well under the outer timeout through a merged runtime or CI-topology change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, github-actions
- Domain
- ci-cd, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100