fix(speculative): the Gemma 4 31B pairing's exactness probe fails both kernels on M3 Ultra, vetoing the burst #1217 just enabled
- Dominant language
- Rust
- Stars
- 467
- Forks
- 54
- Avg merge
- 4h 25m
- Merged PRs (30d)
- 310
Description
## Problem
#1217 measured the batch-capable Gemma 4 31B + bf16 assistant pairing at 1.95x to 2.65x on M3 Ultra and switched `mtp_b1_default` to enable the B = 1 burst for batch-capable targets from Apple GPU generation 15. But that measurement ran at `9e2c6675`, which predates #1258's Gemma exactness probe. On current `main` (`dd21ada4`) the probe for this pairing fails under `qmv_wide` and fails again with it disabled, so the default-env gate declines MTP at serve time on the very host and pairing #1217 enabled by default. The static gate says run, the measured gate says decline, and the decline wins, so the 1.95x to 2.65x is unreachable in a default configuration today.
## Evidence
Measured 2026-08-22 on a Mac Studio M3 Ultra 512 GB, macOS 26.6.1, during the issue #1261 measurement, recorded in `docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md` in PR #1278.
Two independent server boots of `mlxcel-server -m models/gemma-4-31b-it-4bit --model-draft models/gemma-4-31b-it-assistant-bf16 --draft-block-size 4` with `MLXCEL_ENABLE_MTP_B1=1 MLXCEL_MTP_ADAPTIVE=0` both logged:
```
MTP declined: verify block position 0 differs from the single-token chain in 231782 of 524288 logit bytes. Disabling qmv_wide did not make it exact either. Falling back to classic decode.
```
The same host's Gemma 4 12B + 4-bit assistant pairing (block 5) and Qwen 3.8 27B + MTP head pairing (block 3) both pass after the narrow retry, so this is specific to the 31B pairing, not to the host or the gate.
A plausible mechanism is the one #1258's "What this does not settle" section names for the M1 Ultra prose divergence: a narrow-kernel divergence involving the 262144-wide LM head. The 31B's head is 5376 to 262144 where the 12B's is 3840 to 262144, so hidden width may be the discriminator, but that is a hypothesis, not a measurement.
## Candidate directions
To be decided by measurement, not assumed:
1. Probe at other block widths, since exactness is width-dependent and the burst could run at a width whose probe passes.
2. Characterize the divergence at op level for the 31B head shape to confirm or refute the LM-head mechanism.
3. If no exact configuration exists, decide whether `mtp_b1_default`'s generation 15 batch-capable default should stay (it is harmless but misleading, since the exactness gate vetoes it) and update `docs/benchmarks.md`'s 31B rows, which currently imply the speedup ships by default.
4. Re-measure whether `MLXCEL_MTP_ALLOW_INEXACT=1` throughput still matches #1217's rows post-#1258.
## Acceptance criteria
- [ ] The 31B pairing's block-width exactness profile on generation 15 is measured and recorded.
- [ ] Either a default configuration exists in which the pairing engages MTP with byte-identity, or the docs and the `mtp_b1_default` comment state plainly that the pairing requires `MLXCEL_MTP_ALLOW_INEXACT=1` on generation 15+ and what that forfeits.
- [ ] `docs/benchmarks.md`'s 31B speedup rows carry the qualification.
## References
- #1217 (enabled the default)
- #1258 (added the probe)
- #1261 / PR #1278 (where this was found)
- `docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md` (the record)
Contributor guide
Research direction
Start by reproducing the two server boots for the Gemma 4 31B pairing on a generation 15 host, using the command and environment variables in the issue. Read the recorded evidence in docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md and the 31B rows in docs/benchmarks.md, then inspect the mtp_b1_default behavior referenced by #1217 and #1258. Done means the width profile is recorded and the default behavior and benchmark qualification are documented according to the measured result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, rust
- Domain
- ai, documentation, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100