google / google/or-tools

[Bug] RoutingModel segfaults when running multiple solver instances in same process on Linux

Open
#5,076 3 comments 0 reactions 0 assignees View on GitHub
Bug Solver: Routing
Dominant language
C++
Stars
14.1k
Forks
2.5k
Avg merge
8h 39m
Merged PRs (30d)
72

Description

## What version of OR-Tools and what language are you using?

- **Version**: 9.11 ~ 9.15 (tested on multiple versions, all reproduce)
- **Language**: Python 3.11+

## Which solver are you using?

Routing Solver (`pywrapcp.RoutingModel`)

## What operating system (Linux, Windows, ...) and version?

- **Linux** (Ubuntu 22.04 / 24.04, x86_64) — **100% reproducible crash**
- **macOS** (14 Sonoma, Apple Silicon & Intel) — works fine, no crash

## What did you do?

We run two independent `RoutingModel` VRP solvers in the same process:
- **Plan A**: multi-vehicle VRP (`build_collection_routes_vrp`)
- **Plan B**: single-vehicle iterative VRP (`build_collection_routes_vrp_iterative`)

We tried three approaches, all crash on Linux:

1. **Sequential execution** — Plan A completes, then Plan B starts → segfault
2. **Parallel via `threading.Thread`** — both run concurrently → segfault
3. **Parallel via `threading.Thread` with separate DB connections** — same result

Each solver uses its own `RoutingIndexManager`, `RoutingModel`, callbacks, and data — **no shared state at the Python level**.

## What did you expect to see?

Both solvers should complete independently since they use separate `RoutingModel` instances.

## What did you see instead?

Segmentation fault (null dereference at address `0x21d`) when the second `RoutingModel` solver begins execution. The crash occurs inside `_pywrapcp.so`.

- On **Linux**: 100% reproducible
- On **macOS**: Works perfectly — same code, same data, same Python version

## Root cause analysis

The `_pywrapcp.so` shared library contains **C++ global/static state** that gets modified during solver execution. After the first `RoutingModel` solver completes and its resources are freed, the global state retains **dangling pointers**. When the second solver accesses this corrupted global state:

- On **Linux** (`glibc ptmalloc2`): freed memory is aggressively reclaimed via `munmap`, so dangling pointer access → immediate `SIGSEGV`
- On **macOS** (`Apple libmalloc`): freed memory pages tend to remain resident (marked free but not unmapped), so the same dangling pointer reads stale-but-accessible data → **appears to work** (classic undefined behavior)

This is the same class of issue as:
- #1958 (CP-SAT SIGINT handler not thread-safe)
- #2591 (CP-SAT segfault after thousands of sequential invocations)
- #4400 (CP-SAT `FeasibilityJumpSolver` destructor crash on Linux)

But this report is specifically about the **Routing Solver** (`RoutingModel`), which has no documented thread-safety caveats.

## Workaround

Spawn each solver in an **independent subprocess** using `multiprocessing.get_context("spawn")`, so each gets a fresh Python interpreter with clean `_pywrapcp.so` global state:

```python
import multiprocessing as mp

def vrp_plan_subprocess_worker(queue, plan_type, sites, kwargs):
"""Each subprocess gets its own clean _pywrapcp.so instance."""
from ortools.constraint_solver import pywrapcp, routing_enums_pb2
# ... build RoutingModel and solve ...
queue.put(("ok", result))

ctx = mp.get_context("spawn") # MUST be "spawn", not "fork"
qa, qb = ctx.Queue(), ctx.Queue()
pa = ctx.Process(target=vrp_plan_subprocess_worker, args=(qa, "multi", sites, plan_a_kwargs))
pb = ctx.Process(target=vrp_plan_subprocess_worker, args=(qb, "iterative", sites, plan_b_kwargs))
pa.start()
pb.start()
pa.join(timeout=vrp_timeout)
pb.join(timeout=vrp_timeout)
```

**Key points:**
- Must use `"spawn"` context — `"fork"` inherits the parent's already-loaded `_pywrapcp.so` with potentially corrupted state
- `threading.Thread` does **NOT** work — threads share the same process memory space
- Performance overhead is minimal (~1-3s subprocess startup vs 30-60s solver time)
- This also avoids the memory leak from cyclic references (#4753, #4092) since subprocess termination reclaims all memory

## Suggestion

The Routing Solver's thread-safety limitations should be **documented**. Currently, users discover this through production crashes rather than documentation. A note in the [Python Routing reference](https://developers.google.com/optimization/reference/python/constraint_solver/pywrapcp) stating that **"multiple RoutingModel instances should not be used concurrently or sequentially in the same process"** would save significant debugging time.

## Environment

- Python: 3.11
- OR-Tools: 9.11, 9.14, 9.15 (all reproduce on Linux)
- Linux: Ubuntu 22.04 LTS (x86_64), glibc 2.35
- macOS: 14 Sonoma (no crash — same code & data)
- Use case: Production VRP route optimization for waste collection fleet management

## Related issues

- #1958 — CP-SAT multi-thread crash (SIGINT handler)
- #2001 — CP-SAT multi-threaded crash
- #2591 — CP-SAT sequential invocation segfault
- #4400 — CP-SAT occasional segfault on Linux
- #4092 — Memory leak in `AddAtSolutionCallback`
- #4753 — Uncleaned cyclic reference in Python Routing lib (still open)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.