antirez / antirez/ds4

ds4-server with Laguna s2.1 on dgx spark failed in routed expert

Offen
#625 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
22.3k
Forks
2.1k
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
4

Beschreibung

Hi,

I am attempting to run the Laguna model using OpenClaw and Roo Code, but I am encountering errors.

Here are two separate attempts with ds4-server that yielded the same result:

1) ./ds4-server -m gguf/laguna-s-2.1-Q4_K_M.gguf -c 32768
ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings: 62.28 GiB
ds4: CUDA startup model preparation covered 63.56 GiB of tensor spans in 8.281s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 1.57 GiB (raw 1.57 + compressed 0.00) + buffers 0.00 GiB + resident model 63.56 GiB = 65.13 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=1 raw_kv_rows=32768 compressed_kv_rows=512 backend=cuda
0729 09:34:36 ds4-server: context buffers 1608.75 MiB (ctx=32768, backend=cuda, prefill_chunk=1, raw_kv_rows=32768, compressed_kv_rows=512)
ds4: Laguna GPU graph: ctx=32768, prefill=16384, KV 1.57 GiB, scratch 5862.21 MiB
0729 09:34:37 ds4-server: listening on http://127.0.0.1:8000
0729 09:35:49 ds4-server: chat ctx=0..25101:25101 RESPPROTO TOOLS prompt start
ds4: CUDA Laguna routed MoE intermediate quantize launch failed: invalid argument
ds4: Laguna batch prefill failed in routed experts after 1/48 layers

2) ./ds4-server -m gguf/laguna-s-2.1-Q4_K_M.gguf --dflash gguf/laguna-s-2.1-DFlash-Q8_0.gguf
ds4: Linux cuda backend set oom_score_adj=1000
ds4: Laguna DFlash support loaded: gguf/laguna-s-2.1-DFlash-Q8_0.gguf (weights=q8_0, draft=15, p-min=0.40, block=16, cache=512)
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings: 62.28 GiB
ds4: CUDA startup model preparation covered 63.56 GiB of tensor spans in 8.645s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 1.57 GiB (raw 1.57 + compressed 0.00) + buffers 0.00 GiB + resident model 63.56 GiB + support model 1.10 GiB = 66.23 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=1 raw_kv_rows=32768 compressed_kv_rows=512 backend=cuda
0729 09:39:33 ds4-server: context buffers 1608.75 MiB (ctx=32768, backend=cuda, prefill_chunk=1, raw_kv_rows=32768, compressed_kv_rows=512)
ds4: Laguna GPU graph: ctx=32768, prefill=16384, KV 1.57 GiB, scratch 5862.21 MiB
ds4: DFlash graph: block=16, history=512, KV 12.00 MiB, scratch 68.88 MiB
0729 09:39:33 ds4-server: listening on http://127.0.0.1:8000
0729 09:39:39 ds4-server: chat ctx=0..25163:25163 RESPPROTO TOOLS prompt start
ds4: CUDA Laguna routed MoE intermediate quantize launch failed: invalid argument
ds4: Laguna batch prefill failed in routed experts after 1/48 layers

The prompt was a simple "Hallo"

Any ideas to solve ?

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.