antirez / antirez/ds4

GLM-5.3-Flash FP8 Optimized Inference Support

Aperta
#1,015 3 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
22.3k
Fork
2.1k
Merge medio
1g 3h
PR unite (30g)
4

Descrizione

As stated in `docs/MODELS.md` (line 46):
```
| `glm53-fp8` | 305 GiB | Packaged native weights only; inference not implemented
```
inference for `GLM-5.3-Flash` FP8 model is not yet supported in the current `ds4` engine.

I tried simply patching `ds4.c` by routing GLM Q8_0 experts through the existing generic Metal ID kernels and adding Q8_0 coverage to the F32 reference path (see [https://github.com/zzyu17/ds4/commit/61b0d30c30d05cd118335f60e044a8f772225c01](url) for details), which gives the below results for a GLM-5.3-Flash Q8_0 GGUF on M3 Ultra 512G.

Image

The performance is quite good, with ~390 token/s prefill and ~21 token/s generation. However, an optimized (Metal) inference kernel for GLM-5.3-Flash FP8 model should perform even better, compared to the generic Metal kernel patch.

So, looking for the optimized (Metal) inference support for GLM-5.3-Flash FP8 model, eagerly!

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.