antirez / antirez/ds4

GLM --mtp on Metal (server): greedy output diverges from sequential decode, identical requests become nondeterministic, ignore_eos is ignored

Open
#938 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

## Setup

- Hardware: Mac Studio, M3 Ultra (80 GPU cores), 512 GB, Metal backend
- Models: `GLM-5.2-UD-Q2_K_RoutedQ2K.gguf` and full `GLM-5.3-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf` (both rope models, `n_rot == 64`)
- Binary: upstream `main` at `110afdd` plus the one-line gate from PR #936. That fix is a prerequisite for measuring GLM MTP on these models at all — before it, the b0c31af dense-compact flash prefill (#932) corrupts every prompt with ≥ 24 tokens and masks everything below.
- Server: `ds4-server --mtp --mtp-timing`, fresh process with an empty `--kv-disk-dir` per configuration, single session, no SSD streaming
- Requests: OpenAI chat completions, `glm-*-chat` alias, `temperature: 0`, fixed seed, `max_tokens: 1024`, `ignore_eos: true`. Prompts are fixed slices of the tracked `rendered_prompts.txt` corpus: W1 ≈ 956 tokens, W2 ≈ 5836 tokens.

## Finding 1 — MTP greedy output diverges from sequential decode

Greedy speculative decoding must be token-exact against the sequential path. With identical prompts, seed, and sampling, all four comparisons produced different text:

| Model | Workload | no MTP | --mtp | greedy-identical |
|---|---|---:|---:|---|
| GLM-5.2 | W1 | 1024 tok | 415 tok | no |
| GLM-5.2 | W2 | 1024 tok | 105 tok | no |
| GLM-5.3 | W1 | 1024 tok | 81 tok | no |
| GLM-5.3 | W2 | 1024 tok | 1024 tok | no |

The GLM-5.3/W1 and GLM-5.2/W2 rows stop early (finding 3). The GLM-5.3/W2 row runs the full 1024 tokens on both sides and still diverges right after a position where the sequential path has `ignore_eos` suppress the EOS argmax — the MTP side neither commits EOS (it continues) nor picks the same next-best token, so the committed token at that position differs from both the raw argmax and the suppressed argmax.

## Finding 2 — identical requests become nondeterministic under MTP

On one GLM-5.2 + `--mtp` server, the same W1 request sent twice back to back:

- run 1: `gen=415 finish=stop`, text A
- run 2: `gen=222 finish=stop`, text B — same first ~300 chars, then diverges mid-prose (e.g. `"...al massimo delle FLOPS"` vs `"...al massimo delle capacità di calcolo"`)

The second request reuses the live session (rewind to the common prefix, final prompt token re-evaluated — server log shows a 1-token re-prefill). The same replay path without MTP is byte-identical across runs.

Two things we checked that do **not** explain it:

- `ds4_session_rewind()` does reset `glm_mtp_have`, `glm_mtp_rollback_valid`, and `mtp_draft_valid`, so a stale pending draft surviving the rewind is unlikely.
- #905 (unused suffix of a partially consumed MTP block left in the live KV) is adjacent, but its fix is not on `main` at the time of writing and its documented symptom is prefix invalidation, not output divergence. It may still be the underlying mechanism; we could not confirm.

## Finding 3 — `ignore_eos` is ignored on the GLM spec path

Three of four MTP runs ended with `finish=stop` on a stop token despite `ignore_eos: true`; all no-MTP runs reached the 1024-token cap. Code-level cause: the GLM branch of `ds4_session_eval_speculative_argmax_impl` discards the EOS state (`(void)eos_token;`) and `ds4_session_glm_spec_cycle` does not take EOS information at all. The first token still comes from the suppressing argmax, but the verified/drafted second token is a raw argmax — when that is a stop token it gets committed and the server loop stops, where the sequential `argmax_ignoring_eos` path would have suppressed it and continued.

Normal requests want to stop at EOS, so this mostly affects benchmark-style `ignore_eos` clients — but it also means MTP changes stopping behavior relative to the non-MTP server.

## Acceptance and throughput context (for #920 / #676)

`--mtp-timing` cycle lines over the same runs:

- GLM-5.2: 296/446 cycles accepted (66.4%)
- GLM-5.3: 445/737 accepted (60.4%)

Decode throughput (server decode logs, tok/s):

| Model | region | no MTP | --mtp | delta |
|---|---|---:|---:|---:|
| GLM-5.3 | W1 dense | 16.76 | 13.32 / 13.52 | ≈ −20% |
| GLM-5.3 | W2 indexed | 15.97 | 8.83 | ≈ −45% |
| GLM-5.2 | W1 dense | 16.75 | 10.41 / 10.94 | ≈ −37% |
| GLM-5.2 | W2 indexed | 16.00 | 9.03 | ≈ −44% |

At 60–66% acceptance the width-2 cycle costs ≈ 1.8–2.2× a sequential step on this GPU, so MTP is a net loss for these rope models on M3 Ultra even before the correctness findings — consistent with the economics reported in #676 and the verify-cost work in #920 (which shows the opposite sign on M4 Max with GLM-5.3 Flash Q2, so this is model/hardware dependent).

## Repro sketch

```sh
./ds4-server --host 127.0.0.1 --port 8001 --mtp --mtp-timing \
--model gguf/GLM-5.2-UD-Q2_K_RoutedQ2K.gguf --ctx 393216 \
--kv-disk-dir /tmp/fresh-kv --kv-disk-space-mb 2048

# same request twice, compare responses (finding 2), then once more
# without --mtp for the baseline (findings 1 and 3)
curl -s localhost:8001/v1/chat/completions -d '{
"model": "glm-5.2-chat",
"messages": [{"role":"user","content": "<~950 tokens of fixed corpus text>\n\nContinue this document."}],
"temperature": 0, "seed": 1, "max_tokens": 1024, "ignore_eos": true
}'
```

## Related

- #932 / PR #936 — the b0c31af attention regression that previously masked all of the above on rope models
- #905 — unused MTP block suffix in the live KV; same family as finding 2, fix not on `main` yet
- #920 — width-2 verify cost on Metal (throughput context)
- #676 — speculative decoding net-negative economics on Apple silicon (DSpark)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.