docker / docker/model-runner

`unable to load runner: error waiting for runner to be ready: llama.cpp terminated unexpectedly`

Open
#479 17 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
651
Forks
155
PR merge metrics
No merged PRs in 30d

Description

I'm getting this error:

Failed to generate a response: error response: status=500 body=unable to load runner: error waiting for runner to be ready: llama.cpp terminated unexpectedly: llama.cpp exit status: exit status 0xc0000005

I'm with DD on Windows, using WSL2 (ECI disabled in this case).
DMR enabled via DD.
I'm able to run docker model pull commands, docker model list:

MODEL NAME           PARAMETERS  QUANTIZATION    ARCHITECTURE  MODEL ID      CREATED       CONTEXT  SIZE
gemma3               3.88 B      MOSTLY_Q4_K_M   gemma3        a353a8898c9d  2 months ago           2.31 GiB
qwen2.5:0.5B-F16     494.03 M    F16             qwen2         3e1aad67b4cc  7 months ago           942.43 MiB
smollm2:360M-Q4_K_M  361.82 M    IQ2_XXS/Q4_K_M  llama         354bf30d0aa3  8 months ago           256.35 MiB

But when running docker model run ai/qwen2.5:0.5B-F16 "who are you", I'm getting the error described above: unable to load runner: error waiting for runner to be ready: llama.cpp terminated unexpectedly.
Also, docker model status is saying:

Docker Model Runner is running

Status:
llama.cpp: running llama.cpp latest-cpu (sha256:ea16f02ab4b7ce60f05a2cc3d08d2643e53f2c7bb9187c6644fbf108d898739d) version: unknown
vllm: not installed

Looking at docker model logs -f, seeing this:

-------------------------------------------------------------------------------->8
[2025-11-20T20:49:58.776194000Z][inference] Running on system with 32265 MB RAM
[2025-11-20T20:49:58.780206900Z][inference.model-manager] Successfully initialized store
[2025-11-20T20:49:58.780206900Z][inference] 2 backends available
-------------------------------------------------------------------------------->8
[2025-11-20T20:56:13.809536300Z][inference] Running on system with 4094 MB VRAM
[2025-11-20T20:56:13.834988700Z][inference] Running on system with 32265 MB RAM
[2025-11-20T20:56:13.836031600Z][inference.model-manager] Successfully initialized store
[2025-11-20T20:56:13.836578100Z][inference] 2 backends available
[2025-11-20T20:56:17.663784000Z][inference] Reconciling service state on initialization
[2025-11-20T20:56:17.664289000Z][inference] Reconciling service state on settings change
[2025-11-20T20:56:17.664289000Z][inference.inference-llama.cpp] downloadLatestLlamaCpp: latest, cpu, C:\Program Files\Docker\Docker\resources\model-runner\bin, <HOME>\.docker\bin\inference\com.docker.llama-server.exe
[2025-11-20T20:56:18.991409200Z][inference.inference-llama.cpp][W] could not get llama.cpp version: exit status 0xc0000005
[2025-11-20T20:56:18.991409200Z][inference.inference-llama.cpp] failed to ensure latest llama.cpp: bundled llama.cpp version is up to date, no need to update
[2025-11-20T20:56:19.174425300Z][inference.inference-llama.cpp][W] Failed to determine if llama-server is built with GPU support: exit status 0xc0000005
[2025-11-20T20:56:19.174425300Z][inference.inference-llama.cpp] installed llama-server with gpuSupport=false
[2025-11-20T20:56:19.174425300Z][inference][W] Backend installation failed for vllm: not implemented
[2025-11-20T20:56:36.211617900Z][inference.model-manager] Listing available models
[2025-11-20T20:56:36.212180800Z][inference.model-manager] Successfully listed models, count: 0

And this:

[2025-12-02T15:57:10.302145800Z][inference.model-manager] Successfully listed models, count: 3
[2025-12-02T15:57:15.206081100Z][inference.model-manager] Getting model by reference: ai/qwen2.5:0.5B-F16
[2025-12-02T15:57:18.599097500Z][inference.model-manager] Getting model by reference: ai/qwen2.5:0.5B-F16
[2025-12-02T15:57:18.600749200Z][inference.model-manager] Getting model by reference: ai/qwen2.5:0.5B-F16
[2025-12-02T15:57:18.602932000Z][inference.model-manager] Checking model by reference: sha256:3e1aad67b4cc8e3dca660fe65f9f73edb598474284256ffdd9ba460b5b35ff26
[2025-12-02T15:57:23.575350900Z][inference] Loading sha256:3e1aad67b4cc8e3dca660fe65f9f73edb598474284256ffdd9ba460b5b35ff26, which will require 1563 MB RAM and 129 MB VRAM on a system with 32265 MB RAM and 4094 MB VRAM
[2025-12-02T15:57:23.575350900Z][inference] Loading llama.cpp backend runner with model sha256:3e1aad67b4cc8e3dca660fe65f9f73edb598474284256ffdd9ba460b5b35ff26 in completion mode
[2025-12-02T15:57:23.575350900Z][inference.openai-recorder][W] SetConfigForModel called with nil config for model sha256:3e1aad67b4cc8e3dca660fe65f9f73edb598474284256ffdd9ba460b5b35ff26
[2025-12-02T15:57:23.583948900Z][inference.inference-llama.cpp] llama.cpp args: [-ngl 999 --metrics --model C:\\Users\\<USER>\\.docker\\models\\bundles\\sha256\\3e1aad67b4cc8e3dca660fe65f9f73edb598474284256f...[truncated] --host C:\\Users\\<USER>\\AppData\\Local\\Docker\\run\\inference-0.sock --ctx-size 4096 --jinja]
[2025-12-02T15:57:24.117776700Z][inference][W] Backend llama.cpp running model ai/qwen2.5:0.5B-F16 exited with error: llama.cpp terminated unexpectedly: llama.cpp exit status: exit status 0xc0000005
[2025-12-02T15:57:24.577909900Z][inference.model-manager] Getting model by reference: sha256:3e1aad67b4cc8e3dca660fe65f9f73edb598474284256ffdd9ba460b5b35ff26
[2025-12-02T15:57:24.581195900Z][inference.openai-recorder][W] No records found for model: sha256:3e1aad67b4cc8e3dca660fe65f9f73edb598474284256ffdd9ba460b5b35ff26
[2025-12-02T15:57:24.581195900Z][inference][W] Initialization for llama.cpp backend runner with model sha256:3e1aad67b4cc8e3dca660fe65f9f73edb598474284256ffdd9ba460b5b35ff26 in completion mode failed: llama.cpp terminated unexpectedly: llama.cpp exit status: exit status 0xc0000005

More info about my setup:

Processor	Intel(R) Core(TM) Ultra 7 155H (1.40 GHz)
System type	64-bit operating system, x64-based processor

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce with docker model run ai/qwen2.5:0.5B-F16 "who are you" on Docker Desktop with WSL2, then inspect docker model status and docker model logs -f. Start from the llama.cpp startup and exit-status logs shown in the issue; done means the model loads and responds without the runner terminating unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux
Domain
ai, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.