antirez / antirez/ds4

bug to support deepseek harness memory access issue

Abierto
#812 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

hello expert

here is I setup ds4 with L40s I make deepseek harness call into ds4, but I see below error code

```
root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/dpv4harness/ds4# cd /mnt/nvme0n1/wayne/dpv4harness/ds4 && ./ds4-server --cuda --model "$(readlink -f ./ds4flash.gguf)" --ctx 163840 --kv-disk-dir /mnt/nvme0n1/wayne/dpv4harness/ds4-kv --kv-disk-space-mb 655360 --host 0.0.0.0 --port 9090
ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on NVIDIA L40S (sm_89) dev=0
ds4: CUDA (no-copy) registered 80.76 GiB model mapping for multi-tier selective cache
ds4: CUDA preparing model tensor mappings: 80.24 GiB
ds4: CUDA startup model preparation covered 80.76 GiB of tensor spans in 0.034s
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 2.46 GiB (raw 0.36 + compressed 2.10) + buffers 1.25 GiB + resident model 80.76 GiB = 84.47 GiB planned
ds4: memory detail: ctx=163840 prefill_cap=4096 raw_kv_rows=4352 compressed_kv_rows=40962 backend=cuda
0816 04:18:44 ds4-server: context buffers 3797.75 MiB (ctx=163840, backend=cuda, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=40962)
0816 04:18:45 ds4-server: KV disk cache /mnt/nvme0n1/wayne/dpv4harness/ds4-kv (budget=655360 MiB, cross-quant=accept, min=512, cold_max=30000, continued=10000, trim=32, align=2048, hit_half_life=21600s)
0816 04:18:45 ds4-server: listening on http://0.0.0.0:9090
0816 04:19:11 ds4-server: chat ctx=0..9148:9148 TOOLS prompt start
0816 04:19:11 ds4-server: chat ctx=0..9148:9148 TOOLS prefill chunk 0/9148 (0.0%) chunk=0.00 t/s avg=0.00 t/s 0.001s
ds4: CUDA end commands failed: an illegal memory access was encountered
ds4: CUDA synchronize failed: an illegal memory access was encountered
ds4: Metal synchronize after layer-major prefill failure also failed
ds4: CUDA synchronize failed: an illegal memory access was encountered
ds4: Metal synchronize after chunked prefill failure also failed
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.