[BUG] MIP SIGSEGV - Multiple OpenMP runtime conflict

Summary

Solving a MIP problem with ~15K binary variables and ~27K constraints causes a SIGSEGV in libgomp.so.1 after the solver prints "Optimal solution found." The crash occurs during post-solve cleanup — specifically during the destruction of the barrier solver's Cholesky factorization (cuDSS) while OpenMP worker threads are still active.

The crash is reproducible from an MPS file (no custom model-building code needed).

Reproduction

cuopt_sigsegv.mps.zip
The attached MPS file cuopt_sigsegv.mps (2.2 MB, 15,368 binary vars, 26,956 constraints, 69,878 nnz) reproduces the crash:

from cuopt.linear_programming import SolverSettings
from cuopt.linear_programming.problem import Problem

prob = Problem.readMPS("cuopt_sigsegv.mps")
settings = SolverSettings()
settings.set_parameter("time_limit", 20.0)
settings.set_parameter("mip_relative_gap", 0.05)
settings.set_parameter("random_seed", 42)
settings.set_parameter("mip_reliability_branching", 1)
settings.set_parameter("presolve", 1)
prob.solve(settings)  # <-- SIGSEGV here

The problem solves successfully with OMP_NUM_THREADS=1 but crashes with OMP_NUM_THREADS>=2.

GDB Analysis

Program received signal SIGSEGV, Segmentation fault.
#0  0x00007ffefc67f6c2 in ?? () from /lib/x86_64-linux-gnu/libgomp.so.1

Thread 102 (crashed): #0 ?? () from libgomp.so.1
  ... (30+ OpenMP worker threads stopped at same address)

Thread 91 (barrier solver destruction):
  #0 cudssDestroy () from libcudss.so.0
  #1 cuopt::linear_programming::sparse_cholesky_cudss_t<...>::~sparse_cholesky_cudss_t()
  #2 cuopt::linear_programming::iteration_data_t<...>::~iteration_data_t()
  ... (inside barrier solver destructor)

Thread 88 (branch_and_bound):
  #0 std::_Sp_counted_ptr<...>::_M_dispose()
  #1 clique_table_t<...>::~clique_table_t()
  ... (clique table destruction)

The solver prints "Optimal solution found." before crashing, confirming the crash is in the post-solve teardown phase.

Root Cause Analysis

The crash is caused by a multiple OpenMP runtime conflict combined with a destruction-order race condition:

1. Three libgomp instances loaded simultaneously

libcuopt.so -> libgomp-855c301a.so.1.0.0    (bundled by cuOpt)
libcudss_mtlayer_gomp.so.0 -> libgomp.so.1  (system library, loaded by cuDSS via dlopen)
libraft.so -> libgomp-855c301a.so.1.0.0     (bundled by RAFT)

The cuDSS threading layer (libcudss_mtlayer_gomp.so.0) is loaded via dlopen with RTLD_LOCAL and links against the system libgomp.so.1. Meanwhile, libcuopt.so itself links against a bundled libgomp-855c301a.so.1.0.0. These are different copies with separate internal state.

2. Destruction race condition

During MIP solution teardown:

1. The barrier solver's iteration_data_t destructor calls cudssDestroy()
2. cudssDestroy() internally calls dlclose() on the cuDSS library
3. This unloads the cuDSS threading layer, which may affect system libgomp.so.1 state
4. Meanwhile, other threads (branch_and_bound, clique cut threads, OpenMP workers) are still using OpenMP via the system library
5. The system libgomp.so.1 internal state is corrupted -> SIGSEGV

3. Why chain rows trigger it

The flight crew scheduling model includes "chain row" constraints (e.g., start_employee_e <= 1). These increase the problem size and trigger more OpenMP parallelism during solving, which means more threads are active during teardown, making the race condition more likely to manifest.

Workaround

Setting OMP_NUM_THREADS=1 avoids the crash by eliminating the multi-threaded OpenMP race. This is not a production solution.

Suggested Fix

The issue is in the barrier solver's Cholesky factorization cleanup. Possible fixes:

1. Unify OpenMP runtimes: Ensure cuDSS threading layer uses the same libgomp as the main cuOpt library (not the system one). This can be done by setting LD_LIBRARY_PATH or rebuilding cuDSS to bundle its own libgomp.

2. Fix destruction ordering in iteration_data_t: Ensure all OpenMP parallel regions complete and all worker threads are joined before calling cudssDestroy(). The finish_clique_thread() and clique_table_t destruction should happen before barrier solver cleanup.

3. Avoid dlclose in cudssDestroy: If cuDSS unloads its threading layer during destruction, this can corrupt shared OpenMP state. The threading layer should remain loaded for the lifetime of the process.

Environment Details

cuOpt: 26.04.000 (source: v26.02.00a-379-g73601668)
CUDA Toolkit: 13.x
GPU: NVIDIA GeForce RTX 4080 SUPER (SM 8.9)
Driver: 595.58.03
OS: Ubuntu 24.04.4 LTS
GCC: 13.3.0
Python: 3.12.3
Numba: NUMBA_NUM_THREADS=32
GDB: 15.1

cuOpt was built from source with CUDSS threading layer set to libgomp (via CUDSS_MT_LIB_FILE). The build is stripped (no debug symbols).