ml-explore / ml-explore/mlx-c

Xcode 26.5-built MLX-C runtime returns incorrect BF16 gather/RMSNorm/matmul result

Open
#115 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
243
Forks
89
Avg merge
14d 43m
Merged PRs (30d)
2

Description

While porting Apertus to the MLX-based Ollama runner, we observed corrupted output after rebuilding the runtime with Xcode 26.5.
I reduced this to a standalone MLX-C repro with constant tensors and no model/application code.

The same repro passes when MLX-C/MLX is built with Xcode 26.4, but fails when built with Xcode 26.5.

Result Matrix

MLX-C pinned ref fba4470 + Xcode 26.5: FAIL, first=512
MLX-C main + Xcode 26.5:               FAIL, first=512
MLX-C pinned ref fba4470 + Xcode 26.4: PASS, first=1024

Full pinned MLX-C ref:

fba4470b89073180056c9ea46c443051375f7399

The included run used CMAKE_OSX_DEPLOYMENT_TARGET=26.0.

Repro Operation

The repro computes:

BF16 take_axis/gather -> fast_rms_norm -> BF16 matmul

with constant tensors:

embedding:      [131072, 4096] BF16, all 0.5
RMSNorm weight: [4096] BF16, all 1.0
q_proj weight:  [4096, 4096] BF16, all 0.25

Expected:

4096 * (0.5 / sqrt(0.5^2 + 1e-5)) * 0.25 ~= 1023.9795

Observed with Xcode 26.5-built runtime:

first=512.00000000
status=FAIL

Observed with Xcode 26.4-built runtime:

first=1024.00000000
status=PASS

The 512 result is equivalent to the matmul consuming the unnormalized 0.5 input rather than the RMSNorm output.

Toolchain Details

From summary.log:

Xcode 26.4 SDK: 26.4
Metal Toolchain: com.apple.MobileAsset.MetalToolchain-v17.5.188.0.mGsQWM
Result: PASS

Xcode 26.5 SDK: 26.5
Metal Toolchain: com.apple.MobileAsset.MetalToolchain-v17.6.42.0.1BIDXR
Result: FAIL

Both toolchains report:

Apple metal version 32023.883 (metalfe-32023.883)
Target: air64-apple-darwin25.5.0

Reproduction

The attached archive contains:

mlx-c-bug/run.sh
mlx-c-bug/repro.c
mlx-c-bug/README.md
mlx-c-bug/results/20260514-193925/summary.log
per-case repro logs, commit refs, runtime file lists, and checksums

Run:

./mlx-c-bug/run.sh --clean-work

The script clones ml-explore/mlx-c, builds MLX-C/MLX directly with the selected Xcode versions, collects the produced runtime files, compiles repro.c against libmlxc.dylib, and runs the constant tensor check.

Question

Is this expected to be an Xcode/Metal Toolchain regression, or is there a build flag/configuration needed for MLX-C with Xcode 26.5 and deployment target 26.0?

mlx-c-bug-repro-20260514.zip

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached mlx-c-bug/run.sh and README.md, then run ./mlx-c-bug/run.sh --clean-work to reproduce the result. Compare the Xcode 26.4 and 26.5 builds using repro.c and the recorded summary.log and runtime files. Done means identifying whether the failure is an Xcode/Metal Toolchain regression or a required MLX-C build configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, macos
Domain
build-system, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.