deepinsight / deepinsight/insightface

model_zoo.get_model silently drops sess_options → thread oversubscription in CPU-quota containers (50s vs 0.27s inference)

Open
#2,899 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29.7k
Forks
6.1k
PR merge metrics
No merged PRs in 30d

Description

### Environment

- insightface `1.0.1` (PyPI), onnxruntime `1.29.0`, CPU-only (`CPUExecutionProvider`)
- Linux arm64, Docker container with cgroup CPU quota = 4 cores on a 128-core host (cpuset-limited to cores 0-3)
- Model pack: `buffalo_l` (5 ONNX sessions)

### Symptom

`FaceAnalysis` inference was ~**50 seconds** per call (recognize) / ~35s (register) in the container, vs ~0.27s after the workaround below — a ~185x degradation caused purely by thread oversubscription.

`ls /proc//task | wc -l` showed **646 threads**: 5 sessions x 128 threads + main thread. All 640 inference threads were pinned by the container's cpuset onto 4 physical cores, burning the quota on context switches and spin-waits.

### Root cause

`model_zoo.get_model()` accepts arbitrary `**kwargs` but only forwards `providers` / `provider_options` to the session constructor — **`sess_options` is silently dropped**:

```python
# insightface/model_zoo/model_zoo.py
def get_model(name, **kwargs):
...
router = ModelRouter(model_file)
providers = kwargs.get('providers', get_default_providers())
provider_options = kwargs.get('provider_options', get_default_provider_options())
model = router.get_model(providers=providers, provider_options=provider_options)
return model
```

So this intuitive call does nothing, without any warning:

```python
analyzer = FaceAnalysis(name="buffalo_l", root=..., providers=["CPUExecutionProvider"],
sess_options=so) # <-- silently ignored
```

This matters because onnxruntime sizes its intra-op thread pool from the **total system core count** (`sysconf(_SC_NPROCESSORS_ONLN)`), which ignores both cgroup CFS quota and cpuset affinity. In any CPU-limited container this produces heavy oversubscription (see also #2558 and microsoft/onnxruntime#8313). `OMP_NUM_THREADS` does not help either, since the pypi build does not use OpenMP for its thread pool.

### Secondary issue: monkeypatching `onnxruntime.InferenceSession` cannot work

The session is created via the subclass `PickableInferenceSession`, which binds to `onnxruntime.InferenceSession` at import time and is not exported (not in `__all__`, and not reachable from the `insightface.model_zoo` package namespace). Replacing `ort.InferenceSession` after importing insightface has no effect; the only workaround is patching `insightface.model_zoo.model_zoo.PickableInferenceSession` itself (submodule import required).

### Suggested fix

Forward `sess_options` in `get_model()` (and/or accept it explicitly on `FaceAnalysis`), ~3 lines:

```python
def get_model(name, **kwargs):
...
sess_options = kwargs.get('sess_options', None)
model = router.get_model(providers=providers, provider_options=provider_options,
sess_options=sess_options)
```

(or simply pass `**kwargs` through to `ModelRouter.get_model` / `PickableInferenceSession`.)

### Workaround we use in production

```python
import onnxruntime as ort
import insightface.model_zoo.model_zoo as model_zoo

class _PinnedSession(model_zoo.PickableInferenceSession):
def __init__(self, model_path, **kw):
so = ort.SessionOptions()
so.intra_op_num_threads = 4 # container CPU limit
so.inter_op_num_threads = 1
kw["sess_options"] = so
super().__init__(model_path, **kw)

model_zoo.PickableInferenceSession = _PinnedSession
```

Result: 646 → 26 threads, inference 50s → 0.27s on the same container.

Related: #2558 (feature request to expose session options — this report adds the container/CPU-quota failure mode, measurements and root-cause analysis).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.