Would new fused ops be welcome in mx.fast?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 28.5k
- Forks
- 2.3k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 62
Description
Every Llama-family decoder block does this twice per layer:
h = x + residual
y = mx.fast.rms_norm(h, w, eps)
That is two passes over the hidden state. mx.fast.rms_norm is already a tuned kernel, but
added ahead of it materialises h and reads it straight back.
fusing them into one pass reads x and residual, writes h and y, is done.
vLLM ships fused_add_rms_norm for this reason.
I wrote one via mx.fast.metal_kernel to find out whether it is worth anything on Apple silicon. Numbers below are fp16 on an M3 MacBook Air (81 GB/s measured copy bandwidth), median of 40 runs, compared against the current two-call path rather than against naive ops:
| shape | x + residual then mx.fast.rms_norm |
fused | speedup |
|---|---|---|---|
| 1 x 4096 | 0.134 ms | 0.184 ms | 0.73x |
| 512 x 4096 | 0.445 ms | 0.412 ms | 1.08x |
| 2048 x 2048 | 0.704 ms | 0.660 ms | 1.07x |
| 2048 x 4096 | 1.230 ms | 1.054 ms | 1.17x |
| 4096 x 4096 | 2.180 ms | 1.808 ms | 1.21x |
Output matches the two-call path to 7.8e-3 in fp16, which is one rounding of the final store.
Being straight about the shape of this:
it is a prefill-only win.
At a single row it is slower, because at that size the dispatch dominates and there is no bandwidth to save.
So it would want either a row-count threshold or documentation that it is for batched shapes.
Is this something you would want in mx.fast?
I have a working kernel and am happy to open a PR with tests if so, but
I did not want to send API surface you had not asked for. Equally happy to hear that ~1.2x on one op is below the bar.
Measurement setup: https://github.com/Rahu378/hardware-aware-gateway
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing mx.fast.rms_norm entry point and the mx.fast.metal_kernel approach described in the issue. Reproduce the reported batched and single-row benchmarks, then evaluate the API and whether a row-count threshold or documentation is needed. Done means a reviewed fused operation with tests confirming output agreement and its performance tradeoffs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100