ds4-server failed to run with 4xA100/80G (prefill failed)
- Ngôn ngữ chính
- C
- Star
- 22.3k
- Fork
- 2.1k
- Merge trung bình
- 1 ngày 3 giờ
- Pull request đã merge (30 ngày)
- 4
Mô tả
Dear author
The ds4-server failed to work with 4xA100/80G, I have to remove the `--cuda-tensor-parallel` then it works!
run
```sh
make cuda CUDA_ARCH=sm_80 -j8
./ds4-server \
--port 3040 \
-m gguf/ds4f.gguf \
--mtp gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
--dspark \
--cuda --cuda-tensor-parallel \
--gpu-vram auto \
--gpu-devices 0,2,1,3 \
--ctx 204800 \
--batched-session 2
```
logs
```
ds4: GPU config: 4 devices [0,2,1,3] requested, budgets 75,75,75,75 GB; auto=true
ds4: Linux cuda backend set oom_score_adj=1000
ds4: DSpark support model detected: gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf (stages=3 block=5 markov_rank=256 tensors=81 missing=0 invalid=0 metadata_errors=0); use --dspark to enable experimental runtime decode
ds4: CUDA backend initialized on NVIDIA A100 80GB PCIe (sm_80) dev=0
ds4: CUDA backend initialized on NVIDIA A100 80GB PCIe (sm_80) dev=2
ds4: CUDA backend initialized on NVIDIA A100 80GB PCIe (sm_80) dev=1
ds4: CUDA backend initialized on NVIDIA A100 80GB PCIe (sm_80) dev=3
ds4: peer access 0->2 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 0->1 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 0->3 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 2->0 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 2->1 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 2->3 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 1->0 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 1->2 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 1->3 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 3->0 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 3->2 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
ds4: peer access 3->1 FAILED validation at size=4096 iter=0; falling back to pinned-host bounce
multi-GPU layout:
GPU0: layers 0-20 + embedding (41.0 / 72.1 GB)
GPU1: layers 21-42 + output head (42.0 / 72.1 GB)
GPU2: (no transformer layers) (0.0 / 72.1 GB)
GPU3: (no transformer layers) (0.0 / 72.1 GB)
ds4: per-tier graph scratch reserved: 2.76 GiB (pre-subtracted from each GPU budget)
ds4: peer access matrix (validated): 0->1 BOUNCE 0->2 BOUNCE 0->3 BOUNCE 1->0 BOUNCE 1->2 BOUNCE 1->3 BOUNCE 2->0 BOUNCE 2->1 BOUNCE 2->3 BOUNCE 3->0 BOUNCE 3->1 BOUNCE 3->2 BOUNCE
ds4: CUDA (no-copy) registered 153.33 GiB model mapping for multi-tier selective cache
ds4: CUDA decode TP half-resident routed experts enabled ([0,128) home, [128,256) partner); other layer tensors remain replicated
ds4: CUDA output TP stores one vocabulary shard per tier (4 ways)
ds4: CUDA tier 0 (device 0) selective weights: 39.81 GiB in 642 ranges
ds4: CUDA tier 1 (device 2) selective weights: 40.69 GiB in 687 ranges
ds4: CUDA tier 2 (device 1) selective weights: 38.82 GiB in 641 ranges
ds4: CUDA tier 3 (device 3) selective weights: 40.69 GiB in 683 ranges
ds4: DSpark support pack tier=2 free=39.70 GiB budget=35.20 GiB
ds4: DSpark support tensors cached on tier 2 (81 tensors, executor tier)
ds4: DSpark base embedding bucket cached on tier 2 (1 tensors)
0807 22:05:38 ds4-server: context buffers 3682.22 MiB (ctx=204800, backend=cuda, prefill_chunk=2048, raw_kv_rows=2304, compressed_kv_rows=51202)
ds4: CUDA routed MoE expert ownership enabled (half-resident decode and prefill)
ds4: CUDA decode TP enabled: pairing lower-half tiers with upper-half tiers
ds4: DSpark target-hidden capture enabled: layers=40,41,42
0807 22:05:38 ds4-server: KV disk cache ./kvcache (budget=81920 MiB, cross-quant=accept, min=512, cold_max=30000, continued=10000, trim=32, align=2048, hit_half_life=21600s)
0807 22:05:38 ds4-server: listening on http://127.0.0.1:3040
0807 22:05:57 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:05:57 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
0807 22:05:57 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:05:57 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
0807 22:05:58 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:05:58 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
0807 22:05:59 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:05:59 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
0807 22:06:02 ds4-server: chat ctx=0..24541:24541 TOOLS prompt start
0807 22:06:02 ds4-server: chat ctx=0..24541:24541 TOOLS prefill chunk 0/24541 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: gpu layer 0 ffn batch encode failed
ds4: gpu layer-major prefill layer 0 encode failed
...
```
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.