Metal command-buffer limits (50 ops / 50 MB) cost 5–8% decode throughput on launch-heavy models (M3 Ultra measurements)

Device commits a Metal command buffer after max_ops_per_buffer dispatches or when the distinct input buffers referenced exceed max_mb_per_buffer (50 / 50 on Ultra-class devices, backend/metal/device.cpp). For a model whose weight matrices are tens of MB each, the byte rule trips roughly once per layer, so a single-token decode step spans 45–90 command buffers. Each boundary costs about 20 µs of GPU time plus a completion handler on the CPU.

Measured on an M3 Ultra (mlx 0.32.2), GLM-5.3-Flash mixed 4/6-bit (45 layers, MoE + hybrid attention, ~1,600 kernels per token), single-stream decode, same prompts, output hashes compared across arms:

Outputs bit-identical, prefill time unchanged, peak memory unchanged during a cold 20k-token prefill. A synthetic chain of 3,200 tiny dependent kernels goes from 8.8 ms to 7.7 ms per step at 200 ops per buffer.

The same variables on a dense Qwen3.8-27B 8-bit model (about 1,000 large matvecs per step) change nothing (52.8 → 53.0 ms/token short, 71.3 → 72.8 at 20k), so the effect is specific to launch-heavy graphs.

Suggestion: scale the byte limit with the size of the largest input buffers, or raise the Ultra defaults; alternatively document the two variables, which are currently undocumented.

A write-up with the per-segment decode profile behind these numbers follows on my blog; happy to share the profiling driver (flag-gated, repeats a segment on sibling layers via mx.depends) if useful.

Start in backend/metal/device.cpp, where max_ops_per_buffer and max_mb_per_buffer govern command-buffer commits. Reproduce the reported launch-heavy decode measurements and compare them with the dense-model control. Done means selecting and implementing one limit or documentation direction, while confirming output hashes, prefill time, and peak memory remain unchanged.