openvinotoolkit / openvinotoolkit/model_server
[Regression] 2026.3.1-gpu hangs with 200% CPU in sched_yield / libopencl-clang on dynamic shape embeddings (working fine in 2026.3-gpu)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 931
- Forks
- 277
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 68
Description
Describe the bug
In openvino/model_server:2026.3.1-gpu, serving embedding models on Intel iGPU with multiple workers (--rest_workers > 1, e.g. 4 or 16) causes OVMS to hang indefinitely when handling variable-length texts (or concurrent dynamic shape requests).
The container CPU stays pegged at ~200% indefinitely (two threads busy-spinning at 100% each in sched_yield) and stops responding to HTTP requests.
Attaching GDB to the hanging process reveals a spinlock deadlock between libigdrcl.so (Intel OpenCL runtime) and libopencl-clang2.so.16:
- Thread 8 (LWP 96) is stuck inside
libopencl-clang2.so.16(Compile()). - Worker threads (e.g. Thread 22 LWP 82) are stuck in a busy spinloop in
sched_yield()insidelibigdrcl.so.
To Reproduce
Steps to reproduce the behavior:
- Model:
OpenVINO/Qwen3-Embedding-0.6B-int8-ov(or any dynamic shape embedding model). - Launch OVMS on Intel GPU with multiple workers:
docker run --device /dev/dri/renderD128:/dev/dri/renderD128 \ -p 9200:9200 \ openvino/model_server:2026.3.1-gpu \ --model_repository_path /models \ --source_model OpenVINO/Qwen3-Embedding-0.6B-int8-ov \ --task embeddings \ --pooling LAST \ --target_device GPU \ --rest_workers 4 \ --rest_port 9200 \ --model_name qwen3-embedding-0.6b - Send variable-length text requests in quick succession or concurrently:
curl -X POST http://localhost:9200/v3/embeddings \ -H "Content-Type: application/json" \ -d '{"model": "qwen3-embedding-0.6b", "input": "testing variable length dynamic shape sentence"}' - OVMS hangs, container CPU pins at 200%, and requests never return.
Expected behavior
The server should properly synchronize dynamic shape OpenCL compilation without deadlocking or busy-spinning in sched_yield.
GDB Backtrace
GDB backtrace of the spinning process in 2026.3.1-gpu:
Thread 8 (LWP 96 - Compilation):
#0 0x00007fc... in ... from /usr/local/lib/libopencl-clang2.so.16
#1 0x00007fc... in Compile () from /usr/local/lib/libopencl-clang2.so.16
#2 0x00007fc... in ... from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
Thread 22 (LWP 82 - Worker spinning in sched_yield):
#0 0x00007fc... in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fc... in ... from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#2 0x00007fc... in ... from /ovms/lib/libopenvino_intel_gpu_plugin.so
Environment & Hardware Configuration
- Host CPU: 12th Gen Intel Core i5-1235U (10 cores, 12 threads)
- GPU: Intel Iris Xe Graphics (Alder Lake-UP3 GT2, 80 EU, PCI ID
[8086:46a8]) - RAM: 64 GB
- Host OS: Linux 6.6.x (Debian 12)
- Model:
OpenVINO/Qwen3-Embedding-0.6B-int8-ov - Docker Image:
openvino/model_server:2026.3.1-gpu(Broken) vsopenvino/model_server:2026.3-gpu(Working)
Root Cause Analysis & Verified Workarounds
The root trigger is that when multiple worker threads (--rest_workers > 1) handle concurrent embedding requests with variable-length text, multiple threads simultaneously call into the Intel OpenCL driver (libigdrcl.so) and trigger JIT compilation in libopencl-clang2.so.16, resulting in a spinlock deadlock (sched_yield). Furthermore, in 2026.3.1-gpu, once dynamic compilation is triggered, background compiler threads can remain permanently spinning at 200% CPU even after HTTP calls complete.
Workarounds:
- Reverting to
openvino/model_server:2026.3-gpuwith--rest_workers 1and--truncate true: requests execute smoothly at ~40ms latency and CPU drops back to 0.07% immediately upon completion. - Under
2026.3.1-gpu, setting--rest_workers 1and--truncate trueserializes requests at the HTTP layer, butlibopencl-clangthreads may still leak CPU loops after JIT compilation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the hang with the provided Docker command, dynamic-shape embedding requests, multiple --rest_workers, and the 2026.3.1-gpu image; compare it with 2026.3-gpu. Use the supplied GDB backtraces to investigate libigdrcl.so, libopencl-clang2.so.16, and libopenvino_intel_gpu_plugin.so. Done means concurrent requests complete without a deadlock, persistent sched_yield spinning, or a nonresponsive server.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, docker, linux
- Domain
- backend, machine-learning, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100