[xpu-aglinment] Fix GELU erf-tail catastrophic cancellation
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
### 🐛 Describe the bug
Exact GELU on XPU loses the negative tail to catastrophic cancellation. The XPU
SYCL kernel computes the normal CDF term as `0.5 * (1 + erf(x * sqrt(1/2)))`.
For large negative values `erf` approaches `-1`, so `1 + erf(...)` cancels and
the remaining precision is lost.
This was found during XPU alignment for upstream PyTorch PR
https://github.com/pytorch/pytorch/pull/189234. CPU/CUDA/MPS were covered by the
upstream fix, while XPU was explicitly excluded as an out-of-tree backend, so
XPU still exhibits the cancellation.
### Reproducer
```python
import math
import torch
import torch.nn.functional as F
x = torch.arange(-12.0, 12.0, 2**-6, device="xpu", dtype=torch.float32)
xref = x.cpu().double()
tail = xref.abs() >= 4.0
alpha = math.sqrt(0.5)
expected_fwd = 0.5 * xref * torch.erfc(-xref * alpha)
actual_fwd = F.gelu(x).cpu().double()
fwd_error = ((actual_fwd - expected_fwd)[tail] /
expected_fwd[tail].abs().clamp(min=1e-30)).abs().max().item()
grad = torch.ones_like(x)
expected_bwd = (0.5 * torch.erfc(-xref * alpha) +
xref * math.sqrt(2.0 / math.pi) * 0.5 *
torch.exp(-0.5 * xref * xref))
actual_bwd = torch.ops.aten.gelu_backward(grad, x).cpu().double()
bwd_error = ((actual_bwd - expected_bwd)[tail] /
expected_bwd[tail].abs().clamp(min=1e-30)).abs().max().item()
print(f"forward tail relative error: {fwd_error:.3e}")
print(f"backward tail relative error: {bwd_error:.3e}")
assert fwd_error < 1e-3
assert bwd_error < 1e-3
```
### Actual behavior
The exact-GELU forward tail is corrupted by cancellation instead of matching the
stable `erfc` reference:
```text
gelu forward tail relative error (|x|>=4): 1.000e+00
gelu_backward tail relative error (|x|>=4): 3.256e-02
XPU gelu forward: catastrophic cancellation detected (err=1.000e+00)
RESULT: confirmed
```
### Expected behavior
Forward and backward negative-tail relative error stay below `1e-3` for
`|x| >= 4`, matching a float64 `erfc` reference and CPU/CUDA eager semantics.
### Alignment metadata
- Upstream source: https://github.com/pytorch/pytorch/pull/189234
- Source type: upstream PR (CPU/CUDA/MPS fixed; XPU excluded)
- Local XPU result: confirmed
- Routed area: `intel/torch-xpu-ops`, `GeluErfFunctor` / `GeluErfBackwardFunctor` in `src/ATen/native/xpu/sycl/ActivationGeluKernel.cpp`
### Notes
Use the stable expression `0.5 * erfc(-x * sqrt(1/2))` for the CDF term in both
exact-GELU forward and backward. Preserve the PDF term and the tanh-approximate
GELU path. The failure is specific to exact GELU, not the tanh approximation.
A regression test should be added under `test/regressions/` and run with
`pytest` after an editable XPU build; it must fail on the baseline and pass after
the fix.
### Versions
```text
PyTorch version: 2.14.0.dev20260713+xpu
OS: Ubuntu 22.04.5 LTS (x86_64)
Is XPU available: True
Intel GPU: Intel(R) Data Center GPU Max 1100
Relevant path: src/ATen/native/xpu/sycl/ActivationGeluKernel.cpp
```
Contributor guide
Assessment
This issue has not been assessed yet.