microsoft / microsoft/onnxruntime

`import onnxruntime` segfaults in 1.29.0 on containers without `/bin` or `/usr/bin` (works in 1.28.0)

Open
#32,501 1 comment 1 reaction 2 assignees Claimed by @tianleiwu View on GitHub
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

## The issue

`import onnxruntime` crashes with SIGSEGV (exit code 139) in a container image that has no
executables in `/bin` or `/usr/bin` — the normal shape of a hardened or distroless runtime image.
The same image with **onnxruntime 1.28.0** imports fine, and restoring `/bin` + `/usr/bin` makes
1.29.0 import fine, so 1.29.0 appears to execute something at import time and not check whether it
succeeded.

This makes 1.29.0 unusable in shell-less images. We hit it in a Docker image that
must ship without a shell; our only options today are pinning to 1.28.0 or putting a shell back
into the runtime image.

Python-level traceback (via `-X faulthandler`):

```
Fatal Python error: Segmentation fault

Current thread 0x00007f135efd5340 (most recent call first):
File "", line 488 in _call_with_frames_removed
File "", line 1301 in exec_module
...
File "/opt/venv/lib/python3.12/site-packages/onnxruntime/capi/_pybind_state.py", line 32 in
File "/opt/venv/lib/python3.12/site-packages/onnxruntime/__init__.py", line 26 in
File "", line 1 in

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg (total: 2)
```

## To reproduce

```dockerfile
FROM python:3.12-slim AS builder
RUN python -m venv --copies /opt/venv \
&& /opt/venv/bin/pip install --no-cache-dir onnxruntime==1.29.0

FROM python:3.12-slim
COPY --from=builder /opt/venv /opt/venv
# Stand in for a hardened / distroless runtime: interpreter and shared libraries present, but no
# executables in /bin or /usr/bin. The interpreter lives in /usr/local, so it keeps working.
RUN rm -rf /bin /usr/bin
# Exec form: there is no shell left to run a shell-form RUN.
RUN ["/opt/venv/bin/python", "-X", "faulthandler", "-c", "import onnxruntime; print('imported OK', onnxruntime.__version__)"]
```

`docker build` fails at the last step with exit code 139. Changing the pin to
`onnxruntime==1.28.0` makes the same build succeed.

Note: this only reproduces on **real x86-64 hardware**. Under QEMU x86-64 emulation (Docker
Desktop on aarch64) the import succeeds, so a repro attempt on an Apple Silicon machine will look
like a false negative.

## What we ruled out

Each of the following was a separate build against the same image:

- `onnxruntime==1.28.0` — **imports fine**
- Restoring `/bin` + `/usr/bin` from the builder stage — **imports fine**
- Restoring `/etc` — still crashes
- Restoring `/usr/lib/x86_64-linux-gnu` — still crashes
- Restoring all of `/usr/lib` — still crashes
- Restoring glibc locale data (`/usr/lib/locale`) — still crashes
- Running as `USER 0` instead of the image's non-root user — still crashes
- `numpy` pinned to 2.2.6 instead of 2.5.3 — still crashes
- `OPENBLAS_NUM_THREADS=1` — still crashes
- `OMP_NUM_THREADS=1` — still crashes
- `OPENBLAS_MAIN_FREE=1` — still crashes
- `import numpy.linalg` alone, without onnxruntime — imports fine

It is not a missing shared library either: with `LD_DEBUG=libs`, every `find library=` resolves,
and the crash comes *after* `onnxruntime_pybind11_state...so`'s own initializer completes. The last
linker event in the failing run is numpy's `_umath_linalg` init:

```
calling init: .../onnxruntime/capi/onnxruntime_pybind11_state...so <- completes
calling init: .../numpy.libs/libscipy_openblas64_-f48b354e.so
calling init: .../numpy/_core/_multiarray_umath...so
calling init: .../numpy/linalg/_umath_linalg...so <- last event, then SIGSEGV
```

In a working run (1.28.0, or 1.29.0 with `/bin` present) the very next event is:

```
calling init: .../onnxruntime/capi/libonnxruntime_providers_shared.so
```

So the fault window is between the numpy import that ORT's module init triggers and the `dlopen`
of `libonnxruntime_providers_shared.so` — which suggests provider/EP discovery as the place to
look for something newly exec'd in 1.29.0.

So, it blocks 1.29.0 in any hardened or distroless deployment (the workaround is to pin 1.28.0).

## Platform / environment

- **Platform:** Linux
- **OS:** Debian 13 (trixie), glibc 2.41; also reproduced on Debian 12, glibc 2.36
- **Architecture:** X64 (real hardware; not reproducible under emulation)
- **ONNX Runtime installation:** released PyPI package,
`onnxruntime-1.29.0-cp312-cp312-manylinux_2_28_x86_64.whl`
- **ONNX Runtime version:** 1.29.0 (1.28.0 unaffected)
- **ONNX Runtime API:** Python
- **Execution provider:** default CPU
- **Python:** 3.12.14
- **numpy:** 2.5.3 (also reproduced with 2.2.6)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.