bug to support deepseek harness memory access issue
- Lenguaje dominante
- C
- Estrellas
- 22.3k
- Forks
- 2.1k
- Merge medio
- 1 d 3 h
- PR fusionados (30 d)
- 4
Descripción
hello expert
here is I setup ds4 with L40s I make deepseek harness call into ds4, but I see below error code
```
root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/dpv4harness/ds4# cd /mnt/nvme0n1/wayne/dpv4harness/ds4 && ./ds4-server --cuda --model "$(readlink -f ./ds4flash.gguf)" --ctx 163840 --kv-disk-dir /mnt/nvme0n1/wayne/dpv4harness/ds4-kv --kv-disk-space-mb 655360 --host 0.0.0.0 --port 9090
ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on NVIDIA L40S (sm_89) dev=0
ds4: CUDA (no-copy) registered 80.76 GiB model mapping for multi-tier selective cache
ds4: CUDA preparing model tensor mappings: 80.24 GiB
ds4: CUDA startup model preparation covered 80.76 GiB of tensor spans in 0.034s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 2.46 GiB (raw 0.36 + compressed 2.10) + buffers 1.25 GiB + resident model 80.76 GiB = 84.47 GiB planned
ds4: memory detail: ctx=163840 prefill_cap=4096 raw_kv_rows=4352 compressed_kv_rows=40962 backend=cuda
0816 04:18:44 ds4-server: context buffers 3797.75 MiB (ctx=163840, backend=cuda, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=40962)
0816 04:18:45 ds4-server: KV disk cache /mnt/nvme0n1/wayne/dpv4harness/ds4-kv (budget=655360 MiB, cross-quant=accept, min=512, cold_max=30000, continued=10000, trim=32, align=2048, hit_half_life=21600s)
0816 04:18:45 ds4-server: listening on http://0.0.0.0:9090
0816 04:19:11 ds4-server: chat ctx=0..9148:9148 TOOLS prompt start
0816 04:19:11 ds4-server: chat ctx=0..9148:9148 TOOLS prefill chunk 0/9148 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: CUDA end commands failed: an illegal memory access was encountered
ds4: CUDA synchronize failed: an illegal memory access was encountered
ds4: Metal synchronize after layer-major prefill failure also failed
ds4: CUDA synchronize failed: an illegal memory access was encountered
ds4: Metal synchronize after chunked prefill failure also failed
```
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.