[MPS] `var`/`std`/`var_mean` intermittently return wrong values on large reductions (macOS 27)
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
### 🐛 Describe the bug
Found this while running the inductor test suite with `-n 8`, where `test_multilayer_var_mps` fails intermittently.
The test compares a compiled kernel against eager MPS and it's eager that occasionally returns the wrong value, so the corrupt number shows up on the `expected` side.
With `torch.rand`, the expected variance is ~1/12 = 0.08333 but eager sometimes returns values like 0.09327.
Minimal repro:
```python
import torch
import torch.multiprocessing as mp
SHAPE = (10, 3, 352, 352)
WORKERS = 8
ITERS = 50
def worker(wid, out):
bad = 0
for _ in range(ITERS):
a = torch.rand(SHAPE, dtype=torch.float32)
cpu_var = a.double().var(unbiased=True).item()
mps_var = a.to("mps").var().item()
if abs(mps_var - cpu_var) > 1e-4:
bad += 1
print(
f"w{wid} BAD {mps_var:.8f} cpu {cpu_var:.8f} "
f"ratio {mps_var / cpu_var:.4f}",
flush=True,
)
out.put(bad)
if __name__ == "__main__":
mp.set_start_method("spawn")
out = mp.Queue()
procs = [mp.Process(target=worker, args=(i, out)) for i in range(WORKERS)]
for p in procs:
p.start()
total = sum(out.get() for _ in procs)
for p in procs:
p.join()
print(f"bad={total}/{WORKERS * ITERS}")
```
Output I got:
```text
w2 BAD 0.10826389 cpu 0.08332370 ratio 1.2993
w2 BAD 0.09334407 cpu 0.08338290 ratio 1.1195
w5 BAD 0.09999853 cpu 0.08336407 ratio 1.1995
w0 BAD 0.09328175 cpu 0.08333825 ratio 1.1193
bad=4/400
```
It's rare in one process (~1/800 reads) but much easier to trigger with multiple workers / GPU contention.
A `torch.mps.synchronize()` before `.item()` still returns the bad value, so the GPU is writing it.
The wrong values are also surprisingly structured. The same ratios keep showing up across processes: 1.03, 1.10, 1.12, 1.20, 1.30, 1.34, 1.48, 1.60, 1.80, 2.07, 2.46. The excess is roughly an integer or third-integer multiple of ~0.0299, which looks a lot like partial reduction blocks being counted more than once.
What seems to matter is a small trailing dimension. Keeping the element count at 3717120:
| shape | trailing dim | failures |
| ------------------- | -----------: | -------: |
| `(3717120,)` | 3717120 | 0/240 |
| `(30, 123904)` | 123904 | 0/240 |
| `(10, 3, 123904)` | 123904 | 1/240 |
| `(10, 371712)` | 371712 | 0/240 |
| `(10560, 352)` | 352 | 2/240 |
| `(30, 352, 352)` | 352 | 5/240 |
| `(10, 3, 352, 352)` | 352 | 2/240 |
So plain 2D `(10560, 352)` reproduces it too. The individual rates bounce around between runs but the shapes with a 352 trailing dim are consistently the ones that fail.
`std` and `var_mean` have the same problem, while the hand-written Metal reduction path doesn't:
| op | path | failures |
| ---------------------------------- | -------- | -------: |
| `var` | MPSGraph | 14/240 |
| `std` | MPSGraph | 8/240 |
| `var_mean` | MPSGraph | 8/240 |
| manual `mean`/`sub`/`square`/`sum` | Metal | 0/240 |
| `sum` | Metal | 0/240 |
| `mean` | Metal | 0/240 |
| `norm` | Metal | 0/240 |
The remaining `varianceOfTensor` callers in `ReduceOps.mm` are `var`/`std`/`var_mean`.
As a sanity check, I rewrote `std_var_common_impl_mps` to use the existing Metal reduction path instead of `varianceOfTensor`. That completely removes the failure: 0/1920 over 8 workers and `test_mps.py -k "var or std"` passes 276/276.
Doesn't reproduce on macOS 26. Reproduces with both stock `torch==2.14.0` and a source build of `2.15.0a0+gitc5a3ee1` (main at c5a3ee18b93).
### Versions
```
PyTorch version: 2.15.0a0+gitc5a3ee1
Is debug build: False
CUDA used to build PyTorch: None
ROCm SDK used to build PyTorch: N/A
HIP used to build PyTorch: N/A
OS: macOS 27.0 (arm64)
GCC version: Could not collect
Clang version: 21.0.0 (clang-2100.3.20.102)
CMake version: version 4.4.2
Libc version: N/A
Python version: 3.14.3 (main, Feb 3 2026, 15:32:20) [Clang 17.0.0 (clang-1700.6.3.2)] (64-bit runtime)
Python platform: macOS-27.0-arm64-arm-64bit-Mach-O
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A
CPU:
Apple M1 Pro
Versions of relevant libraries:
[pip3] numpy==2.5.2
[pip3] optree==0.20.0
[pip3] torch==2.15.0a0+gitc5a3ee1
[conda] Could not collect
```
cc @kulinseth @malfet @DenisVieriu97 @jhavukainen @aditvenk @Isalia20
Contributor guide
Research direction
Start in ReduceOps.mm at std_var_common_impl_mps and the varianceOfTensor callers for var, std, and var_mean; compare them with the existing manual Metal reduction path. Run the provided multiprocessing reproducer and test_mps.py -k "var or std". Done means the large reductions no longer return intermittent wrong values and the relevant tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100