intel / intel/torch-xpu-ops

[xpu-alignment] XPU exponential_ can return exact zero for float16 at large finite lambda

Open
#5,166 1 comment 0 reactions 1 assignee Claimed by @laifenxiawucha View on GitHub
ai_generated
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

## Upstream source

- PyTorch PR: https://github.com/pytorch/pytorch/pull/195077 (`Fix exponential_ underflowing to exactly 0 for float16`, currently open and unmerged).
- The upstream change adds `c10::cast_no_underflow` and updates the CPU/CUDA kernels, but does not modify the XPU kernel in this repository.

## Observed XPU behavior

For a finite `lambda`, XPU `Tensor.exponential_()` can narrow a positive intermediate value to an exact zero when the output dtype is `float16`.

In the alignment workflow, with 24 million XPU samples per configuration:

```text
dtype lambda exact zeros minimum positive
float16 1 0 4.883e-04
float16 1024 0 4.768e-07
float16 16384 11655 5.960e-08
float16 65504 47262 5.960e-08
bfloat16 all above 0
float32 all above 0
```

An independent local run on an Intel Data Center GPU Max 1100 reproduced the issue. The original 24-million-sample test produced 11,931 zeros at `lambda=16384` and 46,840 zeros at `lambda=65504`. A reduced one-million-sample test also reproduced it across three fixed seeds, while `lambda=1024`, `bfloat16`, and `float32` remained unaffected.

No tested sample was negative or non-finite.

## Root cause

`ExponentialFunctor` in
`src/ATen/native/xpu/sycl/DistributionTemplates.h` calculates the sample in
`accscalar_t`. `DistributionElementwiseKernelFunctor` then writes that value to
a `scalar_t*` output:

```cpp
*out = transform_func_(static_cast((&rand.x)[i]));
```

For sufficiently large finite `lambda`, the positive accumulate-type result is
smaller than the minimum positive `float16` subnormal and the implicit
`accscalar_t -> scalar_t` conversion rounds it to exactly zero.

This is distinct from #4020, which tracks distribution statistical/moment test
failures rather than deterministic float16 narrowing at the XPU kernel output.

## Reproducer

```python
import torch

torch.xpu.manual_seed(0)

for lambd in (1024.0, 16384.0, 65504.0):
x = torch.empty(1_000_000, dtype=torch.float16, device="xpu")
x.exponential_(lambd)
torch.xpu.synchronize()
print(lambd, int((x == 0).sum().item()), x[x > 0].min().item())
```

Expected for each finite `lambda`: zero exact-zero samples. `lambda=inf` is a
separate valid boundary case and should continue to produce exact zero.

## Suggested fix

Use `c10::cast_no_underflow` at the XPU output-narrowing site after the helper
from PyTorch #195077 becomes available. If the XPU fix must land first, use a
minimal local guard at the same narrowing site and replace it with the common
helper later.

Regression coverage should use `lambda >= 16384` for the current XPU kernel;
`lambda=1024` did not reproduce the XPU failure. The fix should preserve the
random stream, unaffected dtypes, and `exponential_(inf) == 0`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.