[Bug] RoutingModel segfaults when running multiple solver instances in same process on Linux
- Dominant language
- C++
- Stars
- 14.1k
- Forks
- 2.5k
- Avg merge
- 8h 39m
- Merged PRs (30d)
- 72
Description
## What version of OR-Tools and what language are you using?
- **Version**: 9.11 ~ 9.15 (tested on multiple versions, all reproduce)
- **Language**: Python 3.11+
## Which solver are you using?
Routing Solver (`pywrapcp.RoutingModel`)
## What operating system (Linux, Windows, ...) and version?
- **Linux** (Ubuntu 22.04 / 24.04, x86_64) — **100% reproducible crash**
- **macOS** (14 Sonoma, Apple Silicon & Intel) — works fine, no crash
## What did you do?
We run two independent `RoutingModel` VRP solvers in the same process:
- **Plan A**: multi-vehicle VRP (`build_collection_routes_vrp`)
- **Plan B**: single-vehicle iterative VRP (`build_collection_routes_vrp_iterative`)
We tried three approaches, all crash on Linux:
1. **Sequential execution** — Plan A completes, then Plan B starts → segfault
2. **Parallel via `threading.Thread`** — both run concurrently → segfault
3. **Parallel via `threading.Thread` with separate DB connections** — same result
Each solver uses its own `RoutingIndexManager`, `RoutingModel`, callbacks, and data — **no shared state at the Python level**.
## What did you expect to see?
Both solvers should complete independently since they use separate `RoutingModel` instances.
## What did you see instead?
Segmentation fault (null dereference at address `0x21d`) when the second `RoutingModel` solver begins execution. The crash occurs inside `_pywrapcp.so`.
- On **Linux**: 100% reproducible
- On **macOS**: Works perfectly — same code, same data, same Python version
## Root cause analysis
The `_pywrapcp.so` shared library contains **C++ global/static state** that gets modified during solver execution. After the first `RoutingModel` solver completes and its resources are freed, the global state retains **dangling pointers**. When the second solver accesses this corrupted global state:
- On **Linux** (`glibc ptmalloc2`): freed memory is aggressively reclaimed via `munmap`, so dangling pointer access → immediate `SIGSEGV`
- On **macOS** (`Apple libmalloc`): freed memory pages tend to remain resident (marked free but not unmapped), so the same dangling pointer reads stale-but-accessible data → **appears to work** (classic undefined behavior)
This is the same class of issue as:
- #1958 (CP-SAT SIGINT handler not thread-safe)
- #2591 (CP-SAT segfault after thousands of sequential invocations)
- #4400 (CP-SAT `FeasibilityJumpSolver` destructor crash on Linux)
But this report is specifically about the **Routing Solver** (`RoutingModel`), which has no documented thread-safety caveats.
## Workaround
Spawn each solver in an **independent subprocess** using `multiprocessing.get_context("spawn")`, so each gets a fresh Python interpreter with clean `_pywrapcp.so` global state:
```python
import multiprocessing as mp
def vrp_plan_subprocess_worker(queue, plan_type, sites, kwargs):
"""Each subprocess gets its own clean _pywrapcp.so instance."""
from ortools.constraint_solver import pywrapcp, routing_enums_pb2
# ... build RoutingModel and solve ...
queue.put(("ok", result))
ctx = mp.get_context("spawn") # MUST be "spawn", not "fork"
qa, qb = ctx.Queue(), ctx.Queue()
pa = ctx.Process(target=vrp_plan_subprocess_worker, args=(qa, "multi", sites, plan_a_kwargs))
pb = ctx.Process(target=vrp_plan_subprocess_worker, args=(qb, "iterative", sites, plan_b_kwargs))
pa.start()
pb.start()
pa.join(timeout=vrp_timeout)
pb.join(timeout=vrp_timeout)
```
**Key points:**
- Must use `"spawn"` context — `"fork"` inherits the parent's already-loaded `_pywrapcp.so` with potentially corrupted state
- `threading.Thread` does **NOT** work — threads share the same process memory space
- Performance overhead is minimal (~1-3s subprocess startup vs 30-60s solver time)
- This also avoids the memory leak from cyclic references (#4753, #4092) since subprocess termination reclaims all memory
## Suggestion
The Routing Solver's thread-safety limitations should be **documented**. Currently, users discover this through production crashes rather than documentation. A note in the [Python Routing reference](https://developers.google.com/optimization/reference/python/constraint_solver/pywrapcp) stating that **"multiple RoutingModel instances should not be used concurrently or sequentially in the same process"** would save significant debugging time.
## Environment
- Python: 3.11
- OR-Tools: 9.11, 9.14, 9.15 (all reproduce on Linux)
- Linux: Ubuntu 22.04 LTS (x86_64), glibc 2.35
- macOS: 14 Sonoma (no crash — same code & data)
- Use case: Production VRP route optimization for waste collection fleet management
## Related issues
- #1958 — CP-SAT multi-thread crash (SIGINT handler)
- #2001 — CP-SAT multi-threaded crash
- #2591 — CP-SAT sequential invocation segfault
- #4400 — CP-SAT occasional segfault on Linux
- #4092 — Memory leak in `AddAtSolutionCallback`
- #4753 — Uncleaned cyclic reference in Python Routing lib (still open)
Contributor guide
Assessment
This issue has not been assessed yet.