antirez / antirez/ds4

ds4-server failed to run with 4xA100/80G (prefill failed)

Đang mở
#735 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
C
Star
22.3k
Fork
2.1k
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
4

Mô tả

Dear author

The ds4-server failed to work with 4xA100/80G, I have to remove the `--cuda-tensor-parallel` then it works!

run
```sh
make cuda CUDA_ARCH=sm_80 -j8

./ds4-server \
--port 3040 \
-m gguf/ds4f.gguf \
--mtp gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
--dspark \
--cuda --cuda-tensor-parallel \
--gpu-vram auto \
--gpu-devices 0,2,1,3 \
--ctx 204800 \
--batched-session 2
```

logs
```
ds4: GPU config: 4 devices [0,2,1,3] requested, budgets 75,75,75,75 GB; auto=true
ds4: Linux cuda backend set oom_score_adj=1000
ds4: DSpark support model detected: gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf (stages=3 block=5 markov_rank=256 tensors=81 missing=0 invalid=0 metadata_errors=0); use --dspark to enable experimental runtime decode
ds4: CUDA backend initialized on NVIDIA A100 80GB PCIe (sm_80) dev=0
ds4: CUDA backend initialized on NVIDIA A100 80GB PCIe (sm_80) dev=2
ds4: CUDA backend initialized on NVIDIA A100 80GB PCIe (sm_80) dev=1
ds4: CUDA backend initialized on NVIDIA A100 80GB PCIe (sm_80) dev=3
ds4: peer access 0->2 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 0->1 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 0->3 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 2->0 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 2->1 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 2->3 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 1->0 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 1->2 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 1->3 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 3->0 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 3->2 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 3->1 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
multi-GPU layout:
GPU0: layers 0-20 + embedding (41.0 / 72.1 GB)
GPU1: layers 21-42 + output head (42.0 / 72.1 GB)
GPU2: (no transformer layers) (0.0 / 72.1 GB)
GPU3: (no transformer layers) (0.0 / 72.1 GB)
ds4: per-tier graph scratch reserved: 2.76 GiB (pre-subtracted from each GPU budget)
ds4: peer access matrix (validated): 0->1 BOUNCE 0->2 BOUNCE 0->3 BOUNCE 1->0 BOUNCE 1->2 BOUNCE 1->3 BOUNCE 2->0 BOUNCE 2->1 BOUNCE 2->3 BOUNCE 3->0 BOUNCE 3->1 BOUNCE 3->2 BOUNCE
ds4: CUDA (no-copy) registered 153.33 GiB model mapping for multi-tier selective cache
ds4: CUDA decode TP half-resident routed experts enabled ([0,128) home, [128,256) partner); other layer tensors remain replicated
ds4: CUDA output TP stores one vocabulary shard per tier (4 ways)
ds4: CUDA tier 0 (device 0) selective weights: 39.81 GiB in 642 ranges
ds4: CUDA tier 1 (device 2) selective weights: 40.69 GiB in 687 ranges
ds4: CUDA tier 2 (device 1) selective weights: 38.82 GiB in 641 ranges
ds4: CUDA tier 3 (device 3) selective weights: 40.69 GiB in 683 ranges
ds4: DSpark support pack tier=2 free=39.70 GiB budget=35.20 GiB
ds4: DSpark support tensors cached on tier 2 (81 tensors, executor tier)
ds4: DSpark base embedding bucket cached on tier 2 (1 tensors)
0807 22:05:38 ds4-server: context buffers 3682.22 MiB (ctx=204800, backend=cuda, prefill_chunk=2048, raw_kv_rows=2304, compressed_kv_rows=51202)
ds4: CUDA routed MoE expert ownership enabled (half-resident decode and prefill)
ds4: CUDA decode TP enabled: pairing lower-half tiers with upper-half tiers
ds4: DSpark target-hidden capture enabled: layers=40,41,42
0807 22:05:38 ds4-server: KV disk cache ./kvcache (budget=81920 MiB, cross-quant=accept, min=512, cold_max=30000, continued=10000, trim=32, align=2048, hit_half_life=21600s)
0807 22:05:38 ds4-server: listening on http://127.0.0.1:3040
0807 22:05:57 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:05:57 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
0807 22:05:57 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:05:57 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
0807 22:05:58 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:05:58 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
0807 22:05:59 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:05:59 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
0807 22:06:02 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:06:02 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
...
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.