antirez / antirez/ds4

Unexpected slow decode and prefill(?) when using --power

Abierto
#465 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

When using the --power flag I saw extremely bad decode speeds ~0.11tok/s.

I tried the extremely naive approach of having a warmup period in order to get a more reliable avg speed and a guard against outliers, which seemed to solve the issue.

This was from an m5 max 128gb `./ds4 -c 30000 --power 50`
```
ds4: metal graph token pos=96 encode=6.150 ms execute=3499.725 ms read=0.010 ms total=3505.885 ms logits=1
ds4: metal graph token pos=97 encode=5.487 ms execute=1018.332 ms read=0.008 ms total=1023.827 ms logits=1
ds4: metal graph token pos=98 encode=5.848 ms execute=1023.379 ms read=0.008 ms total=1029.235 ms logits=1
ds4: prefill: 25.39 t/s, generation: 0.24 t/s
```

https://github.com/antirez/ds4/pull/464

^ for your reference @antirez

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.