pytorch / pytorch/pytorch

torch.nn.functional.softplus returns +Inf for finite float16 input when beta is very small

Open
#187,180 3 comments 0 reactions 0 assignees View on GitHub
bot-triaged module: correctness (silent) module: edge cases module: half module: nn topic: fuzzer triaged
Dominant language
Python
Stars
103k
Forks
29.6k
PR merge metrics
PR metrics pending

Description

### 🐛 Describe the bug

`torch.nn.functional.softplus(x, beta)` on a `float16` input returns `+inf` for an ordinary finite input when `beta` is very small, even though the result is finite in `float32`. softplus is `(1/beta) * log1p(exp(beta * x))`. With `beta = 1e-6` and `x = 1.0`, `exp(1e-6) ~= 1.000001` and `log1p(...) ~= 0.693147`, so the value is `0.693147 / 1e-6 ~= 693147.7`. That is finite in fp32 but exceeds the fp16 max of `65504`; the reduced-floating-type kernel widens to fp32 for the math and then does a final `static_cast(...)` back to fp16 with no range check, overflowing to `+inf`.

The linear stability branch (`beta * x > threshold`, default `20`) never fires here because `beta * x = 1e-6`. This is distinct from the large-`beta` exp-overflow case: here the whole fp32 computation is finite and only the narrowing cast overflows.

```python
import torch
import torch.nn.functional as F

x16 = torch.tensor([[1.0]], dtype=torch.float16) # finite input
print("fp16:", F.softplus(x16, beta=1e-6).item())
print("fp32:", F.softplus(x16.float(), beta=1e-6).item())
print("fp16 max:", torch.finfo(torch.float16).max)
```

Observed:

```
fp16: inf
fp32: 693147.6875
fp16 max: 65504.0
```

Expected:

Compute in fp32 and clamp to `finfo(float16).max` before the narrowing cast (or raise on overflow), rather than silently returning `+inf` for a finite input.

### Versions

```
PyTorch version: 2.5.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.3 LTS (x86_64)
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version: 18.1.3 (1ubuntu1)
CMake version: version 3.28.3
Libc version: glibc-2.39

Python version: 3.10.20 (main, Mar 11 2026, 17:46:40) [GCC 14.3.0] (64-bit runtime)
Python platform: Linux-6.8.0-124-generic-x86_64-with-glibc2.39
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 SUPER
Nvidia driver version: 535.309.01
cuDNN version: Could not collect
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU(s): 20
Model name: 12th Gen Intel(R) Core(TM) i7-12700
Thread(s) per core: 2
Core(s) per socket: 12

Versions of relevant libraries:
[pip3] numpy==2.2.6
[pip3] torch==2.5.1+cu121
[pip3] triton==3.1.0
```

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.