Docker Model Runner: `docker model bench` leaves the loaded llama.cpp model returning only `?` tokens (CUDA, GTX 1650)

Environment

Reproduction A — the bench

docker model pull ai/qwen3:4b-instruct-2507-q4_K_M
docker model bench ai/qwen3:4b-instruct-2507-q4_K_M --concurrency 1,2 --duration 30s --json

Observed:

• 9 of 18 requests fail with request failed during benchmark: request failed with status 500: {"error":{"code":500,"message":"Context size has been exceeded.","type":"server_error"}} (2 at concurrency 1, 7 at concurrency 2).
• The 9 requests that succeed stop at 120–137 tokens (a normal 100-word summary), so the model was producing coherent output at least part of the time.
• Bench JSON afterwards: TPS 6.03 at c=1 and 3.21 at c=2, because failed requests' wall time is counted while their tokens are not.
• After the bench, every request to the still-loaded model returns only ? (0x3F) up to max_tokens, finish_reason: "length", never EOS, for example:

curl -s http://localhost:12434/engines/llama.cpp/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"ai/qwen3:4b-instruct-2507-q4_K_M",
  "messages":[{"role":"user","content":"Reply with the single word OK."}],
  "max_tokens":8,"temperature":0.2}'
→ {"content":"????????","finish_reason":"length", ...}

The same happens with temperature: 0, with a system prompt, and with the model addressed by tag, by docker.io/ prefix, or by @sha256: digest. A second model on the same runner (ai/nomic-embed-text-v1.5:137M-F16, embeddings) keeps working. docker model status still reports the runner healthy.

Reproduction B — a forced-continuation streaming request (no bench)

On a freshly loaded, verified-healthy model:

curl -s -N http://localhost:12434/engines/llama.cpp/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"ai/qwen3:4b-instruct-2507-q4_K_M",
  "messages":[{"role":"user","content":"Write a comprehensive 100 word summary on whales and their impact on society."}],
  "max_tokens":8192,"temperature":0.2,"ignore_eos":true,"stream":true}'

Observed: 4071 content deltas; deltas 1–6 are Wh|ales| play| a| vital| role, deltas 7–4071 are all ?. Stream ends with finish_reason: "length" (context full at 4096). Every subsequent request returns only ?, exactly as after the bench.

What does NOT reproduce it (each on a healthy model, each followed by an "OK" probe that passed)

• Non-streaming requests of any kind, including the same ignore_eos + max_tokens 8192 request without stream (4071 coherent tokens, 0 ?, context filled to 4096, model healthy afterwards).
• Streaming with max_tokens 8 or 160, and streaming with max_tokens 8192 but without ignore_eos (model stops at ~125 tokens).
• Client-cancelled streams (connection dropped after 4 s, ~150 deltas), single and two concurrent.
• Two concurrent bounded requests (121/124 tokens, 36 tok/s each).

So: ordinary streaming, cancellation, concurrency and a full context are harmless; the bench and forced-continuation streaming poison the loaded model. The exact request shape docker model bench sends was not reproduced from the API; the bench itself is the reliable reproducer.

Recovery

docker model unload --all, or the runner's idle unload (observed 251 s and 281 s after the last request). Reload after that takes < 2 s (GGUF in page cache) and the model answers normally again ("OK", "4", coherent 120–130-token summaries at ~40 tok/s).

Health probe used

{"messages":[{"role":"user","content":"Reply with the single word OK."}],"max_tokens":8,"temperature":0.2} → healthy iff content is OK and finish_reason is stop.

Start with the docker model bench entry point and the runner's /engines/llama.cpp/v1/chat/completions endpoint; compare their request and streaming paths with the llama-server logs showing context exhaustion and decode() failed. Reproduce the bench and forced-continuation cases, then verify that subsequent health probes return OK without unloading the model while the existing bounded-streaming cases remain healthy.