ml-explore / ml-explore/mlx

Would new fused ops be welcome in mx.fast?

Open
#4,259 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
C++
Stars
28.5k
Forks
2.3k
Avg merge
3d 8h
Merged PRs (30d)
62

Description

Every Llama-family decoder block does this twice per layer:

h = x + residual
y = mx.fast.rms_norm(h, w, eps)

That is two passes over the hidden state. mx.fast.rms_norm is already a tuned kernel, but
added ahead of it materialises h and reads it straight back.
fusing them into one pass reads x and residual, writes h and y, is done.
vLLM ships fused_add_rms_norm for this reason.

I wrote one via mx.fast.metal_kernel to find out whether it is worth anything on Apple silicon. Numbers below are fp16 on an M3 MacBook Air (81 GB/s measured copy bandwidth), median of 40 runs, compared against the current two-call path rather than against naive ops:

shape x + residual then mx.fast.rms_norm fused speedup
1 x 4096 0.134 ms 0.184 ms 0.73x
512 x 4096 0.445 ms 0.412 ms 1.08x
2048 x 2048 0.704 ms 0.660 ms 1.07x
2048 x 4096 1.230 ms 1.054 ms 1.17x
4096 x 4096 2.180 ms 1.808 ms 1.21x

Output matches the two-call path to 7.8e-3 in fp16, which is one rounding of the final store.

Being straight about the shape of this:
it is a prefill-only win.
At a single row it is slower, because at that size the dispatch dominates and there is no bandwidth to save.

So it would want either a row-count threshold or documentation that it is for batched shapes.

Is this something you would want in mx.fast?
I have a working kernel and am happy to open a PR with tests if so, but
I did not want to send API surface you had not asked for. Equally happy to hear that ~1.2x on one op is below the bar.

Measurement setup: https://github.com/Rahu378/hardware-aware-gateway

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing mx.fast.rms_norm entry point and the mx.fast.metal_kernel approach described in the issue. Reproduce the reported batched and single-row benchmarks, then evaluate the API and whether a row-count threshold or documentation is needed. Done means a reviewed fused operation with tests confirming output agreement and its performance tradeoffs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.