NVIDIA / NVIDIA/cuopt

[BUG] ROUTING_TEST CVRPTW_Retail/4 SIGABRT — "Cost should improve" assertion in execute_vrp_moves

Open
#1,276 5 comments 0 reactions 1 assignee View on GitHub

@aliceb-nv is already working on this.

Since May 21, 2026.

awaiting response bug
Dominant language
Cuda
Stars
1k
Forks
233
Avg merge
4d 4h
Merged PRs (30d)
95

Description

Describe the bug

ROUTING_TEST crashed with SIGABRT on the level0_retail/retail_float_test_t.CVRPTW_Retail/4 case. The cuopt_assert invariant on the aggregate cost change after a batch of VRP local-search moves did not hold:

ROUTING_TEST: cpp/src/routing/local_search/vrp/vrp_execute.cu:466:
bool cuopt::routing::detail::execute_vrp_moves(
    solution_t<i_t, f_t, REQUEST>&, move_candidates_t<i_t, f_t>&, i_t)
    [with i_t = int; f_t = float;
     cuopt::routing::request_t REQUEST = cuopt::routing::request_t::VRP]:
Assertion `cost_before - cost_after > EPSILON && "Cost should improve!"' failed.

./ci/run_ctests.sh: line 86: 4820 Aborted (core dumped) "${gt}" --gtest_output="xml:${xml_file}" "$@"
CRASH: ROUTING_TEST died from SIGABRT (exit code 134)

The assertion fires after a batch of accepted VRP moves is executed: the planner predicted a cost decrease, but the post-execution measured cost decrease was not greater than EPSILON. This indicates either stale move-candidate data, conflicting moves applied together, or a predicted/actual delta mismatch.

Where

  • File: cpp/src/routing/local_search/vrp/vrp_execute.cu:466
  • Function: cuopt::routing::detail::execute_vrp_moves<int, float, VRP>
  • Test: level0_retail/retail_float_test_t.CVRPTW_Retail/4
  • Binary: ROUTING_TEST (debug build — assertion is inside cuopt_func_call / cuopt_assert)

Reproducer / observed run

  • PR: #1269 (CI-only change — does not touch routing code)
  • Run: actions/runs/26237894930
  • Job: conda-cpp-tests / 13.2.0, 3.13, amd64, ubuntu24.04, rtxpro6000, latest-driver, latest-deps (job 77225276444)
  • Head SHA: 371edafa (branch pull-request/1269, base release/26.06)
  • Surrounding tests CVRPTW_Retail/0..3 passed; only /4 aborted.

Expected behavior

execute_vrp_moves should only accept and execute moves whose net post-execution cost change matches the predicted delta within EPSILON. The assertion should hold for every batch on every accepted test instance.

Environment

  • CI runner: rtxpro6000 (NVIDIA RTX PRO 6000 Blackwell Server Edition, 94.97 GiB VRAM)
  • CPU: INTEL(R) XEON(R) GOLD 5512U, 16 threads
  • CUDA: 13.2
  • Driver: latest-driver
  • Deps: latest-deps
  • Build: conda, ubuntu24.04, amd64, Python 3.13
  • Branch under test: release/26.06

Related issues

  • #1221 (closed 2026-05-18) — similar CVRPTW_Retail SIGABRT, but on case /18 and root-caused to route-priority-sort out-of-bounds (fixed by 1d85aed6). Current failure is on /4 and at a different assertion site, so likely a distinct root cause that survived that fix.
  • #866 (closed 2026-05-18) — same family of cost-invariant assertion but at local_search.cu:315 ("Cost mismatch after a move"), case /18. Cross-link for context.
  • #402 (closed 2025-11-20) — older instance of the same "Cost should improve" assertion family.

Additional context

  • PR #1269 modifies only .github/workflows/* and ci/release/update-version.sh, so the failure cannot have been caused by its changes. Either:
    1. The test is flaky on this hardware/CUDA combination, or
    2. A latent bug exists on release/26.06 that this run happened to expose.
  • If flaky: consider whether cuopt_assert should be relaxed for the aggregated-batch case, or the failing instance saved for offline reproduction.
  • If a real regression: bisect release/26.06 since #1221's fix landed (1d85aed6 on 2026-05-18) to identify the change that exposed it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.