[CUDA] GLM-5.2/DeepseekV32 mxfp4 forward numerically wrong on Blackwell (sm_120/121/110), correct on Metal — individual ops fine, composed forward diverges ~25%

Summary

The GLM-5.2 (mlx-community/GLM-5.2-mxfp4, model_type: glm_moe_dsa, DeepseekV32-style MLA + 256-expert MoE) forward is numerically wrong on Blackwell CUDA but correct on Metal.

Running the same 4-layer slice of the model (3 dense + 1 MoE/DSA layer) with the real mxfp4 weights loaded from disk, on the same inputs, comparing GPU vs CPU:

(Same pattern on GB10 sm_121 and Jetson Thor sm_110 in the live cluster.) Over 78 layers this compounds into degenerate logits — distributed inference across a mixed Metal+Blackwell ring produces pure garbage ("!" repeated), while a standard int4 model (Qwen3) on the identical ring produces coherent text.

The puzzle: individual ops are fine, the composed forward is not

On the same sm_120 device, GPU-vs-CPU for the building blocks is all within quantization noise:

• mx.quantized_matmul(..., mode='mxfp4') across many shapes (2048×6144, 6144×6144, 1536×6144, 512×6144, 6144×512, 154880×6144, …): 0.010–0.031
• mx.fast.rope: 0.0014 · mx.fast.scaled_dot_product_attention (+causal): 0.003–0.004 · mx.fast.rms_norm: 0.0000

Yet the composed DeepseekV32 dense-layer forward diverges 0.22–0.25 on Blackwell (and only 0.033 on Metal). So it is not any single op in isolation — it is Blackwell-specific behavior of the composed graph (possibly error amplification through the deep MLA attention chain, a CUDA-graph precision issue, or an MLA op I haven't isolated that only misbehaves in-model).

Environment

• mlx 0.32.0.dev20260717+7a1d4f5, source-built with -DMLX_BUILD_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120/121/110.
• RTX 5090 (sm_120, CUDA 12.8, WSL2); GB10 Spark (sm_121, CUDA 13.0); Jetson Thor (sm_110, JetPack 7.1).
• Metal reference: M5 Max / M4 Max.

I have standalone repro scripts (load 4 real GLM layers, forward GPU vs CPU) and can run any instrumented build or narrower repro on the Blackwell hardware. Happy to help bisect which op in the MLA/DeepseekV32 path diverges.

Related: #3876 (distributed barrier deadlock on the same cluster/arch).