ds4-server with Laguna s2.1 on dgx spark failed in routed expert
- Dominant language
- C
- Stars
- 22.3k
- Forks
- 2.1k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
Hi,
I am attempting to run the Laguna model using OpenClaw and Roo Code, but I am encountering errors.
Here are two separate attempts with ds4-server that yielded the same result:
1) ./ds4-server -m gguf/laguna-s-2.1-Q4_K_M.gguf -c 32768
ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings: 62.28 GiB
ds4: CUDA startup model preparation covered 63.56 GiB of tensor spans in 8.281s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 1.57 GiB (raw 1.57 + compressed 0.00) + buffers 0.00 GiB + resident model 63.56 GiB = 65.13 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=1 raw_kv_rows=32768 compressed_kv_rows=512 backend=cuda
0729 09:34:36 ds4-server: context buffers 1608.75 MiB (ctx=32768, backend=cuda, prefill_chunk=1, raw_kv_rows=32768, compressed_kv_rows=512)
ds4: Laguna GPU graph: ctx=32768, prefill=16384, KV 1.57 GiB, scratch 5862.21 MiB
0729 09:34:37 ds4-server: listening on http://127.0.0.1:8000
0729 09:35:49 ds4-server: chat ctx=0..25101:25101 RESPPROTO TOOLS prompt start
ds4: CUDA Laguna routed MoE intermediate quantize launch failed: invalid argument
ds4: Laguna batch prefill failed in routed experts after 1/48 layers
2) ./ds4-server -m gguf/laguna-s-2.1-Q4_K_M.gguf --dflash gguf/laguna-s-2.1-DFlash-Q8_0.gguf
ds4: Linux cuda backend set oom_score_adj=1000
ds4: Laguna DFlash support loaded: gguf/laguna-s-2.1-DFlash-Q8_0.gguf (weights=q8_0, draft=15, p-min=0.40, block=16, cache=512)
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings: 62.28 GiB
ds4: CUDA startup model preparation covered 63.56 GiB of tensor spans in 8.645s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 1.57 GiB (raw 1.57 + compressed 0.00) + buffers 0.00 GiB + resident model 63.56 GiB + support model 1.10 GiB = 66.23 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=1 raw_kv_rows=32768 compressed_kv_rows=512 backend=cuda
0729 09:39:33 ds4-server: context buffers 1608.75 MiB (ctx=32768, backend=cuda, prefill_chunk=1, raw_kv_rows=32768, compressed_kv_rows=512)
ds4: Laguna GPU graph: ctx=32768, prefill=16384, KV 1.57 GiB, scratch 5862.21 MiB
ds4: DFlash graph: block=16, history=512, KV 12.00 MiB, scratch 68.88 MiB
0729 09:39:33 ds4-server: listening on http://127.0.0.1:8000
0729 09:39:39 ds4-server: chat ctx=0..25163:25163 RESPPROTO TOOLS prompt start
ds4: CUDA Laguna routed MoE intermediate quantize launch failed: invalid argument
ds4: Laguna batch prefill failed in routed experts after 1/48 layers
The prompt was a simple "Hallo"
Any ideas to solve ?
Contributor guide
Assessment
This issue has not been assessed yet.