antirez / antirez/ds4

GLM-5.3-Flash FP8 Optimized Inference Support

Open
#1,015 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

As stated in `docs/MODELS.md` (line 46):
```
| `glm53-fp8` | 305 GiB | Packaged native weights only; inference not implemented
```
inference for `GLM-5.3-Flash` FP8 model is not yet supported in the current `ds4` engine.

I tried simply patching `ds4.c` by routing GLM Q8_0 experts through the existing generic Metal ID kernels and adding Q8_0 coverage to the F32 reference path (see [https://github.com/zzyu17/ds4/commit/61b0d30c30d05cd118335f60e044a8f772225c01](url) for details), which gives the below results for a GLM-5.3-Flash Q8_0 GGUF on M3 Ultra 512G.

Image

The performance is quite good, with ~390 token/s prefill and ~21 token/s generation. However, an optimized (Metal) inference kernel for GLM-5.3-Flash FP8 model should perform even better, compared to the generic Metal kernel patch.

So, looking for the optimized (Metal) inference support for GLM-5.3-Flash FP8 model, eagerly!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.