pytorch / pytorch/pytorch

torch.exp2 second derivative overflows to inf at 1024.0 although the float64 value is finite

Open
#192,408 1 comment 0 reactions 0 assignees View on GitHub
bot-triaged module: double backwards module: edge cases triaged
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

### 🐛 Describe the bug

`torch.autograd` returns `inf` for the second derivative of `torch.exp2` at a `float64` input where the mathematically correct second derivative is finite and representable in `float64`.

The second derivative of `exp2(x)` is `(ln(2) ** 2) * 2^x`. For `x = 1024.0`, the true second derivative is approximately `8.637070847446594e+307`, which is below the maximum finite `float64` value. However, PyTorch returns `inf`.

This issue reproduces on CPU, so CUDA availability does not affect the result.

```python
import torch

torch.set_default_dtype(torch.float64)

x = torch.tensor(1024.0, requires_grad=True)

y = torch.exp2(x)

g1, = torch.autograd.grad(y, x, create_graph=True)
g2, = torch.autograd.grad(g1, x)

actual = g2
expected = torch.tensor(8.637070847446594e+307, dtype=torch.float64)

print("actual:")
print(actual)
print("expected:")
print(expected)
```

### Actual vs expected result

```text
actual:
tensor(inf)

expected:
tensor(8.6371e+307)
```

The discrepancy occurs in the second-order backward pass. Although the forward value `torch.exp2(1024.0)` overflows, the mathematical second derivative `(ln(2) ** 2) * 2^1024` is still finite in `float64`, so the second derivative should not be `inf`.

### Versions

```text
PyTorch version: 2.12.0+cu130
Is debug build: False
CUDA used to build PyTorch: 13.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.3 LTS (x86_64)
Python version: 3.13.13 | packaged by Anaconda, Inc.
Python platform: Linux x86_64

Is CUDA available: False
GPU models:
GPU 0: NVIDIA RTX 6000 Ada Generation
GPU 1: NVIDIA RTX 6000 Ada Generation
Nvidia driver version: 570.211.01

CPU:
Model name: AMD Ryzen Threadripper PRO 7985WX 64-Cores
CPU(s): 128

Versions of relevant libraries:
[pip3] numpy==2.4.6
[pip3] torch==2.12.0
[pip3] triton==3.7.0
```

Contributor guide

Open the contributing guide

Research direction

Start by running the provided CPU reproduction with float64 and inspect the torch.exp2 autograd implementation, focusing on the second-order backward pass. The issue names no source file or test path, so trace the operation from its Python entry point into the relevant implementation. Done means the second derivative at 1024.0 is finite and matches the expected value, with regression coverage for this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.