Mixed q2-q4 GGUF + <code>--ssd-streaming</code> on ROCm: prefill fails below ~10 prompt tokens ("selected expert id -1")

(helped myself with Claude for the bug report but the grunt work to isolate it is human. Salvatore, keep it up! you're a real pathfinder and i'd be happy to invite you to dinner or drinks if I'm ever in Sicily)

Summary

On Strix Halo (gfx1151), the mixed Layers37-42Q4KExperts Flash GGUF
fails under --ssd-streaming whenever the token batch is small. The
router returns -1 for every expert slot.

Prompts of 10 tokens or more work. Prompts of 9 tokens or fewer fail
immediately at prefill. Decode — which is one token at a time — fails the
same way once generation starts, even after a long prompt prefills fine.

The plain q2 GGUF works normally with the same flags. The mixed GGUF works
normally without --ssd-streaming.

The QA regression test in QA_BEFORE_RELEASES.md passes on my machine,
because it uses a ~950-token prompt — above the threshold.

Environment

• Minisforum MS-S1 Max, AMD Ryzen AI MAX+ 395, Radeon 8060S (gfx1151), 128 GB


• Fedora 44, kernel 7.1.10-200.fc44.x86_64


• ds4 built from main on 2026-08-29


• No custom kernel parameters. GTT aperture reports 112 GiB.


• Model: DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed-0731.gguf

Fedora needs extra link flags to build (see #626):

make strix-halo -j$(nproc) \

  ROCM_CFLAGS="-O3 -ffast-math -g -fno-finite-math-only -pthread -D__HIP_PLATFORM_AMD__ -Wno-unused-command-line-argument --offload-arch=gfx1151 -no-pie" \

  ROCM_LDLIBS="-lm -pthread -lhipblas -lhipblaslt -lamdhip64"

Minimal reproduction

M=./gguf/DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed-0731.gguf


./ds4 --rocm -m $M --ssd-streaming --ssd-streaming-cache-experts 50GB \

  --ctx 4096 --tokens 1 --nothink -p "hi"

Output:

ds4: rocm SSD streaming total expert budget 50.00 GiB = 3.38 GiB prefill headroom + 46.62 GiB dynamic cache (7073 experts, 6.75 MiB each)

ds4: SSD streaming mixed-precision model: 6/43 routed layers off the slab size class will bypass the expert cache and read experts via mapped model views

...

processing 10 input tokens: 1/10 (10.0%)

ds4: Metal streaming async selected expert id -1 is outside 0..256 at layer 3

ds4: prompt processing failed: rocm prefill failed

The threshold

Prompt files of word repeated N times, run with
--ctx 4096 --tokens 1 --nothink --prompt-file:

words | raw tokens (--dump-tokens) | result
-- | -- | --
8 | 9 | FAIL
9 | 10 | ok
10 | 11 | ok
12 | 13 | ok
20 | 21 | ok
400 | ~401 | ok

Token counts are the raw prompt only; the real runs add chat markers on
top. The boundary was stable across repeated runs.

The threshold does not move with the expert cache budget. 8 words
fails identically at --ssd-streaming-cache-experts 16GB and at 50GB.

It does not depend on context size. Fails the same at --ctx 4096 and
--ctx 393216.

It does not depend on thinking mode. Fails with and without
--nothink.

It does not depend on how the prompt is supplied. Fails the same with
-p "hi" and with --prompt-file containing hi.

Decode fails the same way

With a prompt above the threshold, prefill succeeds and then generation
fails once decoding starts:

./ds4 --rocm -m $M --ssd-streaming --ssd-streaming-cache-experts 50GB \

  --ctx 393216 --tokens 200 --prompt-file /tmp/ds4_600tok_prompt.txt

processing 949 input tokens: 949/949 (100.0%)

1.ds4: ROCm streaming selected expert id -1 outside 0..256 (layer=38 slot=0 selected=[-1,-1,-1,-1,-1,-1,-1,-1])

ds4: decode failed: rocm decode failed

Decode processes one token at a time, so this may be the same small-batch
problem rather than a second bug. The reported layer differs between runs
(3 during prefill, 38 during decode), so it does not look confined to the
37-42 Q4 range.

Controls

Plain q2 GGUF, same streaming flags — works.
...IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf runs normally
at --ctx 400000 with a 50GB expert cache, including short prompts. This
is my everyday setup.

Mixed GGUF, no streaming — works.

./ds4-server --rocm -m $M --ctx 393216 --port 8000

Answers short requests correctly. No expert id -1 messages.

The QA regression test passes. The command in QA_BEFORE_RELEASES.md
uses a ~600-token prompt file, which is well above the threshold, so it
does not exercise this case:

processing 949 input tokens: 949/949 (100.0%)

ds4: prefill: 19.78 t/s, generation: 6161.66 t/s

Via ds4-server

Same failure through the HTTP API, which is how I originally hit it. A
5-token and an 84-token request both failed; the trace shows
finish: error and generated_tokens: 0 for both.

Possibly related

The Trosfy/spark-ds4f fork describes a similar-shaped problem on CUDA:
q2-q4 imatrix GGUFs failed prefill with cuda prefill failed while pure
q2/IQ2_XXS worked, because Q4_K prefill was guarded off in
routed_moe_launch leaving working kernels unreachable. Different backend
and a different error string, but the same "mixed quant fails, pure q2
works" pattern in the routed-MoE path.

Also worth noting: the failing message says "Metal streaming async" while
the run used --rocm on an AMD GPU, so that code path appears shared
between backends.

What I can do

I have the model file and can rerun anything on this hardware. Happy to
test patches, add instrumentation, or narrow the threshold further.