lablup / lablup/mlxcel

fix(speculative): the Gemma 4 31B pairing's exactness probe fails both kernels on M3 Ultra, vetoing the burst #1217 just enabled

Open
#1,279 1 comment 0 reactions 0 assignees View on GitHub
area:inference platform:macos priority:high status:ready type:bug
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

## Problem

#1217 measured the batch-capable Gemma 4 31B + bf16 assistant pairing at 1.95x to 2.65x on M3 Ultra and switched `mtp_b1_default` to enable the B = 1 burst for batch-capable targets from Apple GPU generation 15. But that measurement ran at `9e2c6675`, which predates #1258's Gemma exactness probe. On current `main` (`dd21ada4`) the probe for this pairing fails under `qmv_wide` and fails again with it disabled, so the default-env gate declines MTP at serve time on the very host and pairing #1217 enabled by default. The static gate says run, the measured gate says decline, and the decline wins, so the 1.95x to 2.65x is unreachable in a default configuration today.

## Evidence

Measured 2026-08-22 on a Mac Studio M3 Ultra 512 GB, macOS 26.6.1, during the issue #1261 measurement, recorded in `docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md` in PR #1278.

Two independent server boots of `mlxcel-server -m models/gemma-4-31b-it-4bit --model-draft models/gemma-4-31b-it-assistant-bf16 --draft-block-size 4` with `MLXCEL_ENABLE_MTP_B1=1 MLXCEL_MTP_ADAPTIVE=0` both logged:

```
MTP declined: verify block position 0 differs from the single-token chain in 231782 of 524288 logit bytes. Disabling qmv_wide did not make it exact either. Falling back to classic decode.
```

The same host's Gemma 4 12B + 4-bit assistant pairing (block 5) and Qwen 3.8 27B + MTP head pairing (block 3) both pass after the narrow retry, so this is specific to the 31B pairing, not to the host or the gate.

A plausible mechanism is the one #1258's "What this does not settle" section names for the M1 Ultra prose divergence: a narrow-kernel divergence involving the 262144-wide LM head. The 31B's head is 5376 to 262144 where the 12B's is 3840 to 262144, so hidden width may be the discriminator, but that is a hypothesis, not a measurement.

## Candidate directions

To be decided by measurement, not assumed:

1. Probe at other block widths, since exactness is width-dependent and the burst could run at a width whose probe passes.
2. Characterize the divergence at op level for the 31B head shape to confirm or refute the LM-head mechanism.
3. If no exact configuration exists, decide whether `mtp_b1_default`'s generation 15 batch-capable default should stay (it is harmless but misleading, since the exactness gate vetoes it) and update `docs/benchmarks.md`'s 31B rows, which currently imply the speedup ships by default.
4. Re-measure whether `MLXCEL_MTP_ALLOW_INEXACT=1` throughput still matches #1217's rows post-#1258.

## Acceptance criteria

- [ ] The 31B pairing's block-width exactness profile on generation 15 is measured and recorded.
- [ ] Either a default configuration exists in which the pairing engages MTP with byte-identity, or the docs and the `mtp_b1_default` comment state plainly that the pairing requires `MLXCEL_MTP_ALLOW_INEXACT=1` on generation 15+ and what that forfeits.
- [ ] `docs/benchmarks.md`'s 31B speedup rows carry the qualification.

## References

- #1217 (enabled the default)
- #1258 (added the probe)
- #1261 / PR #1278 (where this was found)
- `docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md` (the record)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two server boots for the Gemma 4 31B pairing on a generation 15 host, using the command and environment variables in the issue. Read the recorded evidence in docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md and the 31B rows in docs/benchmarks.md, then inspect the mtp_b1_default behavior referenced by #1217 and #1258. Done means the width profile is recorded and the default behavior and benchmark qualification are documented according to the measured result.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, rust
Domain
ai, documentation, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.