intel / intel/torch-xpu-ops

`torch.compile` fails with `PassManager::run failed` when `sort` is fused with `cumsum` on XPU

Open
#5,100 2 comments 0 reactions 1 assignee Claimed by @jiqing-feng View on GitHub
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

## Describe the bug

When a `torch.sort` output feeds directly into `torch.cumsum` inside a compiled region, Inductor
fuses both into a single Triton kernel that contains `triton_helpers.sort_with_index` followed by
`tl.associative_scan`. The Intel Triton backend fails to compile that kernel:

```
File "triton/backends/intel/compiler.py", line 456, in make_ttgir
pm.run(mod, 'make_ttgir')
RuntimeError: PassManager::run failed
```

which surfaces as `torch._inductor.exc.InductorError`.

Either op alone compiles fine. Only the fused combination fails. The same code compiles and runs on CUDA.

## To Reproduce

```python
import torch

if torch.xpu.is_available():
device = "xpu"
elif torch.cuda.is_available():
device = "cuda"
else:
device = "cpu"

print(f"torch {torch.__version__}, device {device}")

def sort_then_cumsum(x):
sorted_x, idx = torch.sort(x, dim=-1, descending=False)
cum = torch.cumsum(sorted_x, dim=-1)
mask = cum - sorted_x <= 0.1
return torch.scatter(input=torch.zeros_like(mask), dim=-1, index=idx, src=mask)

def sort_only(x):
sorted_x, idx = torch.sort(x, dim=-1, descending=False)
return sorted_x, idx

def cumsum_only(x):
return torch.cumsum(x, dim=-1)

x = torch.randn(2, 16, device=device)

for name, fn in [("cumsum only", cumsum_only), ("sort only", sort_only), ("sort + cumsum", sort_then_cumsum)]:
try:
torch.compile(fn, fullgraph=True)(x)
print(f"{name:<16} OK")
except Exception as e: # noqa: BLE001
print(f"{name:<16} FAIL: {type(e).__name__}: {str(e).splitlines()[0]}")
```

XPU:

```
torch 2.15.0.dev20260823+xpu, device xpu
cumsum only OK
sort only OK
sort + cumsum FAIL: InductorError: RuntimeError: PassManager::run failed
```

CUDA:

```
torch 2.13.0a0+8145d630e8.nv26.06, device cuda
cumsum only OK
sort only OK
sort + cumsum OK
```

## Expected behavior

`sort + cumsum` compiles on XPU, as it does on CUDA.

## Additional context

The generated Triton kernel puts both primitives in one body:

```python
tmp6, tmp7, = triton_helpers.sort_with_index(tmp4, tmp5, rnumel, 1, stable=False, descending=False)
tmp8 = tmp6.to(tl.float32)
tmp9 = tl.broadcast_to(tmp8, [XBLOCK, R0_BLOCK])
tmp10, = tl.associative_scan((tmp9,), 1, _triton_helper_fn_add0)
```

Inserting `.clone()` or `.contiguous()` between the two ops does not help — Inductor still fuses them.
The only workaround we found is to split the sort so the scan cannot be fused into the sort kernel:

```python
sorted_indices = torch.argsort(x, dim=-1, descending=False)
sorted_x = torch.gather(x, -1, sorted_indices)
cum = torch.cumsum(sorted_x, dim=-1)
```

This pattern appears in real code, e.g. the entropy-bound sampler in Hugging Face `transformers`
(`DiffusionGemma`), where it breaks compiled generation on XPU.

## Versions

- torch: `2.15.0.dev20260823+xpu`
- device: Intel Arc Pro B60
- driver: `1.15.38646+6`
- OS: Linux

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.