antirez / antirez/ds4

GLM-5.3-Flash FP8 Optimized Inference Support

Abierto
#1,015 3 comentarios 1 reacción 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

As stated in `docs/MODELS.md` (line 46):
```
| `glm53-fp8` | 305 GiB | Packaged native weights only; inference not implemented
```
inference for `GLM-5.3-Flash` FP8 model is not yet supported in the current `ds4` engine.

I tried simply patching `ds4.c` by routing GLM Q8_0 experts through the existing generic Metal ID kernels and adding Q8_0 coverage to the F32 reference path (see [https://github.com/zzyu17/ds4/commit/61b0d30c30d05cd118335f60e044a8f772225c01](url) for details), which gives the below results for a GLM-5.3-Flash Q8_0 GGUF on M3 Ultra 512G.

Image

The performance is quite good, with ~390 token/s prefill and ~21 token/s generation. However, an optimized (Metal) inference kernel for GLM-5.3-Flash FP8 model should perform even better, compared to the generic Metal kernel patch.

So, looking for the optimized (Metal) inference support for GLM-5.3-Flash FP8 model, eagerly!

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.