openvinotoolkit / openvinotoolkit/model_server

[Regression] 2026.3.1-gpu hangs with 200% CPU in sched_yield / libopencl-clang on dynamic shape embeddings (working fine in 2026.3-gpu)

Open
#4,510 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
931
Forks
277
Avg merge
2d 13h
Merged PRs (30d)
68

Description

Describe the bug
In openvino/model_server:2026.3.1-gpu, serving embedding models on Intel iGPU with multiple workers (--rest_workers > 1, e.g. 4 or 16) causes OVMS to hang indefinitely when handling variable-length texts (or concurrent dynamic shape requests).

The container CPU stays pegged at ~200% indefinitely (two threads busy-spinning at 100% each in sched_yield) and stops responding to HTTP requests.

Attaching GDB to the hanging process reveals a spinlock deadlock between libigdrcl.so (Intel OpenCL runtime) and libopencl-clang2.so.16:

  • Thread 8 (LWP 96) is stuck inside libopencl-clang2.so.16 (Compile()).
  • Worker threads (e.g. Thread 22 LWP 82) are stuck in a busy spinloop in sched_yield() inside libigdrcl.so.

To Reproduce
Steps to reproduce the behavior:

  1. Model: OpenVINO/Qwen3-Embedding-0.6B-int8-ov (or any dynamic shape embedding model).
  2. Launch OVMS on Intel GPU with multiple workers:
    docker run --device /dev/dri/renderD128:/dev/dri/renderD128 \
      -p 9200:9200 \
      openvino/model_server:2026.3.1-gpu \
      --model_repository_path /models \
      --source_model OpenVINO/Qwen3-Embedding-0.6B-int8-ov \
      --task embeddings \
      --pooling LAST \
      --target_device GPU \
      --rest_workers 4 \
      --rest_port 9200 \
      --model_name qwen3-embedding-0.6b
    
  3. Send variable-length text requests in quick succession or concurrently:
    curl -X POST http://localhost:9200/v3/embeddings \
      -H "Content-Type: application/json" \
      -d '{"model": "qwen3-embedding-0.6b", "input": "testing variable length dynamic shape sentence"}'
    
  4. OVMS hangs, container CPU pins at 200%, and requests never return.

Expected behavior
The server should properly synchronize dynamic shape OpenCL compilation without deadlocking or busy-spinning in sched_yield.

GDB Backtrace
GDB backtrace of the spinning process in 2026.3.1-gpu:

Thread 8 (LWP 96 - Compilation):

#0  0x00007fc... in ... from /usr/local/lib/libopencl-clang2.so.16
#1  0x00007fc... in Compile () from /usr/local/lib/libopencl-clang2.so.16
#2  0x00007fc... in ... from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so

Thread 22 (LWP 82 - Worker spinning in sched_yield):

#0  0x00007fc... in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fc... in ... from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#2  0x00007fc... in ... from /ovms/lib/libopenvino_intel_gpu_plugin.so

Environment & Hardware Configuration

  1. Host CPU: 12th Gen Intel Core i5-1235U (10 cores, 12 threads)
  2. GPU: Intel Iris Xe Graphics (Alder Lake-UP3 GT2, 80 EU, PCI ID [8086:46a8])
  3. RAM: 64 GB
  4. Host OS: Linux 6.6.x (Debian 12)
  5. Model: OpenVINO/Qwen3-Embedding-0.6B-int8-ov
  6. Docker Image: openvino/model_server:2026.3.1-gpu (Broken) vs openvino/model_server:2026.3-gpu (Working)

Root Cause Analysis & Verified Workarounds
The root trigger is that when multiple worker threads (--rest_workers > 1) handle concurrent embedding requests with variable-length text, multiple threads simultaneously call into the Intel OpenCL driver (libigdrcl.so) and trigger JIT compilation in libopencl-clang2.so.16, resulting in a spinlock deadlock (sched_yield). Furthermore, in 2026.3.1-gpu, once dynamic compilation is triggered, background compiler threads can remain permanently spinning at 200% CPU even after HTTP calls complete.

Workarounds:

  1. Reverting to openvino/model_server:2026.3-gpu with --rest_workers 1 and --truncate true: requests execute smoothly at ~40ms latency and CPU drops back to 0.07% immediately upon completion.
  2. Under 2026.3.1-gpu, setting --rest_workers 1 and --truncate true serializes requests at the HTTP layer, but libopencl-clang threads may still leak CPU loops after JIT compilation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the hang with the provided Docker command, dynamic-shape embedding requests, multiple --rest_workers, and the 2026.3.1-gpu image; compare it with 2026.3-gpu. Use the supplied GDB backtraces to investigate libigdrcl.so, libopencl-clang2.so.16, and libopenvino_intel_gpu_plugin.so. Done means concurrent requests complete without a deadlock, persistent sched_yield spinning, or a nonresponsive server.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, docker, linux
Domain
backend, machine-learning, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.