antirez / antirez/ds4

ds4-server with Laguna s2.1 on dgx spark failed in routed expert

Đang mở
#625 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
C
Star
22.3k
Fork
2.1k
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
4

Mô tả

Hi,

I am attempting to run the Laguna model using OpenClaw and Roo Code, but I am encountering errors.

Here are two separate attempts with ds4-server that yielded the same result:

1) ./ds4-server -m gguf/laguna-s-2.1-Q4_K_M.gguf -c 32768
ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings: 62.28 GiB
ds4: CUDA startup model preparation covered 63.56 GiB of tensor spans in 8.281s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 1.57 GiB (raw 1.57 + compressed 0.00) + buffers 0.00 GiB + resident model 63.56 GiB = 65.13 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=1 raw_kv_rows=32768 compressed_kv_rows=512 backend=cuda
0729 09:34:36 ds4-server: context buffers 1608.75 MiB (ctx=32768, backend=cuda, prefill_chunk=1, raw_kv_rows=32768, compressed_kv_rows=512)
ds4: Laguna GPU graph: ctx=32768, prefill=16384, KV 1.57 GiB, scratch 5862.21 MiB
0729 09:34:37 ds4-server: listening on http://127.0.0.1:8000
0729 09:35:49 ds4-server: chat ctx=0..25101:25101 RESPPROTO TOOLS prompt start
ds4: CUDA Laguna routed MoE intermediate quantize launch failed: invalid argument
ds4: Laguna batch prefill failed in routed experts after 1/48 layers

2) ./ds4-server -m gguf/laguna-s-2.1-Q4_K_M.gguf --dflash gguf/laguna-s-2.1-DFlash-Q8_0.gguf
ds4: Linux cuda backend set oom_score_adj=1000
ds4: Laguna DFlash support loaded: gguf/laguna-s-2.1-DFlash-Q8_0.gguf (weights=q8_0, draft=15, p-min=0.40, block=16, cache=512)
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings: 62.28 GiB
ds4: CUDA startup model preparation covered 63.56 GiB of tensor spans in 8.645s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 1.57 GiB (raw 1.57 + compressed 0.00) + buffers 0.00 GiB + resident model 63.56 GiB + support model 1.10 GiB = 66.23 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=1 raw_kv_rows=32768 compressed_kv_rows=512 backend=cuda
0729 09:39:33 ds4-server: context buffers 1608.75 MiB (ctx=32768, backend=cuda, prefill_chunk=1, raw_kv_rows=32768, compressed_kv_rows=512)
ds4: Laguna GPU graph: ctx=32768, prefill=16384, KV 1.57 GiB, scratch 5862.21 MiB
ds4: DFlash graph: block=16, history=512, KV 12.00 MiB, scratch 68.88 MiB
0729 09:39:33 ds4-server: listening on http://127.0.0.1:8000
0729 09:39:39 ds4-server: chat ctx=0..25163:25163 RESPPROTO TOOLS prompt start
ds4: CUDA Laguna routed MoE intermediate quantize launch failed: invalid argument
ds4: Laguna batch prefill failed in routed experts after 1/48 layers

The prompt was a simple "Hallo"

Any ideas to solve ?

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.