pytorch / pytorch/pytorch

[MPS] `var`/`std`/`var_mean` intermittently return wrong values on large reductions (macOS 27)

Open
#197,235 0 comments 0 reactions 0 assignees View on GitHub
bot-triaged module: correctness (silent) module: mps module: regression release triage triage review
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

### 🐛 Describe the bug

Found this while running the inductor test suite with `-n 8`, where `test_multilayer_var_mps` fails intermittently.

The test compares a compiled kernel against eager MPS and it's eager that occasionally returns the wrong value, so the corrupt number shows up on the `expected` side.

With `torch.rand`, the expected variance is ~1/12 = 0.08333 but eager sometimes returns values like 0.09327.

Minimal repro:

```python
import torch
import torch.multiprocessing as mp

SHAPE = (10, 3, 352, 352)
WORKERS = 8
ITERS = 50

def worker(wid, out):
bad = 0
for _ in range(ITERS):
a = torch.rand(SHAPE, dtype=torch.float32)
cpu_var = a.double().var(unbiased=True).item()
mps_var = a.to("mps").var().item()
if abs(mps_var - cpu_var) > 1e-4:
bad += 1
print(
f"w{wid} BAD {mps_var:.8f} cpu {cpu_var:.8f} "
f"ratio {mps_var / cpu_var:.4f}",
flush=True,
)
out.put(bad)

if __name__ == "__main__":
mp.set_start_method("spawn")
out = mp.Queue()
procs = [mp.Process(target=worker, args=(i, out)) for i in range(WORKERS)]
for p in procs:
p.start()
total = sum(out.get() for _ in procs)
for p in procs:
p.join()
print(f"bad={total}/{WORKERS * ITERS}")
```

Output I got:

```text
w2 BAD 0.10826389 cpu 0.08332370 ratio 1.2993
w2 BAD 0.09334407 cpu 0.08338290 ratio 1.1195
w5 BAD 0.09999853 cpu 0.08336407 ratio 1.1995
w0 BAD 0.09328175 cpu 0.08333825 ratio 1.1193
bad=4/400
```

It's rare in one process (~1/800 reads) but much easier to trigger with multiple workers / GPU contention.

A `torch.mps.synchronize()` before `.item()` still returns the bad value, so the GPU is writing it.

The wrong values are also surprisingly structured. The same ratios keep showing up across processes: 1.03, 1.10, 1.12, 1.20, 1.30, 1.34, 1.48, 1.60, 1.80, 2.07, 2.46. The excess is roughly an integer or third-integer multiple of ~0.0299, which looks a lot like partial reduction blocks being counted more than once.

What seems to matter is a small trailing dimension. Keeping the element count at 3717120:

| shape | trailing dim | failures |
| ------------------- | -----------: | -------: |
| `(3717120,)` | 3717120 | 0/240 |
| `(30, 123904)` | 123904 | 0/240 |
| `(10, 3, 123904)` | 123904 | 1/240 |
| `(10, 371712)` | 371712 | 0/240 |
| `(10560, 352)` | 352 | 2/240 |
| `(30, 352, 352)` | 352 | 5/240 |
| `(10, 3, 352, 352)` | 352 | 2/240 |

So plain 2D `(10560, 352)` reproduces it too. The individual rates bounce around between runs but the shapes with a 352 trailing dim are consistently the ones that fail.

`std` and `var_mean` have the same problem, while the hand-written Metal reduction path doesn't:

| op | path | failures |
| ---------------------------------- | -------- | -------: |
| `var` | MPSGraph | 14/240 |
| `std` | MPSGraph | 8/240 |
| `var_mean` | MPSGraph | 8/240 |
| manual `mean`/`sub`/`square`/`sum` | Metal | 0/240 |
| `sum` | Metal | 0/240 |
| `mean` | Metal | 0/240 |
| `norm` | Metal | 0/240 |

The remaining `varianceOfTensor` callers in `ReduceOps.mm` are `var`/`std`/`var_mean`.

As a sanity check, I rewrote `std_var_common_impl_mps` to use the existing Metal reduction path instead of `varianceOfTensor`. That completely removes the failure: 0/1920 over 8 workers and `test_mps.py -k "var or std"` passes 276/276.

Doesn't reproduce on macOS 26. Reproduces with both stock `torch==2.14.0` and a source build of `2.15.0a0+gitc5a3ee1` (main at c5a3ee18b93).

### Versions

```
PyTorch version: 2.15.0a0+gitc5a3ee1
Is debug build: False
CUDA used to build PyTorch: None
ROCm SDK used to build PyTorch: N/A
HIP used to build PyTorch: N/A

OS: macOS 27.0 (arm64)
GCC version: Could not collect
Clang version: 21.0.0 (clang-2100.3.20.102)
CMake version: version 4.4.2
Libc version: N/A

Python version: 3.14.3 (main, Feb 3 2026, 15:32:20) [Clang 17.0.0 (clang-1700.6.3.2)] (64-bit runtime)
Python platform: macOS-27.0-arm64-arm-64bit-Mach-O
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==2.5.2
[pip3] optree==0.20.0
[pip3] torch==2.15.0a0+gitc5a3ee1
[conda] Could not collect
```

cc @kulinseth @malfet @DenisVieriu97 @jhavukainen @aditvenk @Isalia20

Contributor guide

Open the contributing guide

Research direction

Start in ReduceOps.mm at std_var_common_impl_mps and the varianceOfTensor callers for var, std, and var_mean; compare them with the existing manual Metal reduction path. Run the provided multiprocessing reproducer and test_mps.py -k "var or std". Done means the large reductions no longer return intermittent wrong values and the relevant tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.