intel / intel/torch-xpu-ops

[xpu-aglinment] Fix GELU erf-tail catastrophic cancellation

Open
#4,753 0 comments 0 reactions 1 assignee Claimed by @laifenxiawucha View on GitHub
ai_generated
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

### 🐛 Describe the bug

Exact GELU on XPU loses the negative tail to catastrophic cancellation. The XPU
SYCL kernel computes the normal CDF term as `0.5 * (1 + erf(x * sqrt(1/2)))`.
For large negative values `erf` approaches `-1`, so `1 + erf(...)` cancels and
the remaining precision is lost.

This was found during XPU alignment for upstream PyTorch PR
https://github.com/pytorch/pytorch/pull/189234. CPU/CUDA/MPS were covered by the
upstream fix, while XPU was explicitly excluded as an out-of-tree backend, so
XPU still exhibits the cancellation.

### Reproducer

```python
import math
import torch
import torch.nn.functional as F

x = torch.arange(-12.0, 12.0, 2**-6, device="xpu", dtype=torch.float32)
xref = x.cpu().double()
tail = xref.abs() >= 4.0
alpha = math.sqrt(0.5)

expected_fwd = 0.5 * xref * torch.erfc(-xref * alpha)
actual_fwd = F.gelu(x).cpu().double()
fwd_error = ((actual_fwd - expected_fwd)[tail] /
expected_fwd[tail].abs().clamp(min=1e-30)).abs().max().item()

grad = torch.ones_like(x)
expected_bwd = (0.5 * torch.erfc(-xref * alpha) +
xref * math.sqrt(2.0 / math.pi) * 0.5 *
torch.exp(-0.5 * xref * xref))
actual_bwd = torch.ops.aten.gelu_backward(grad, x).cpu().double()
bwd_error = ((actual_bwd - expected_bwd)[tail] /
expected_bwd[tail].abs().clamp(min=1e-30)).abs().max().item()

print(f"forward tail relative error: {fwd_error:.3e}")
print(f"backward tail relative error: {bwd_error:.3e}")
assert fwd_error < 1e-3
assert bwd_error < 1e-3
```

### Actual behavior

The exact-GELU forward tail is corrupted by cancellation instead of matching the
stable `erfc` reference:

```text
gelu forward tail relative error (|x|>=4): 1.000e+00
gelu_backward tail relative error (|x|>=4): 3.256e-02
XPU gelu forward: catastrophic cancellation detected (err=1.000e+00)
RESULT: confirmed
```

### Expected behavior

Forward and backward negative-tail relative error stay below `1e-3` for
`|x| >= 4`, matching a float64 `erfc` reference and CPU/CUDA eager semantics.

### Alignment metadata

- Upstream source: https://github.com/pytorch/pytorch/pull/189234
- Source type: upstream PR (CPU/CUDA/MPS fixed; XPU excluded)
- Local XPU result: confirmed
- Routed area: `intel/torch-xpu-ops`, `GeluErfFunctor` / `GeluErfBackwardFunctor` in `src/ATen/native/xpu/sycl/ActivationGeluKernel.cpp`

### Notes

Use the stable expression `0.5 * erfc(-x * sqrt(1/2))` for the CDF term in both
exact-GELU forward and backward. Preserve the PDF term and the tanh-approximate
GELU path. The failure is specific to exact GELU, not the tanh approximation.

A regression test should be added under `test/regressions/` and run with
`pytest` after an editable XPU build; it must fail on the baseline and pass after
the fix.

### Versions

```text
PyTorch version: 2.14.0.dev20260713+xpu
OS: Ubuntu 22.04.5 LTS (x86_64)
Is XPU available: True
Intel GPU: Intel(R) Data Center GPU Max 1100
Relevant path: src/ATen/native/xpu/sycl/ActivationGeluKernel.cpp
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.