pytorch / pytorch/pytorch

torch.cdist returns +Inf for small p (p-norm overflow, docstring admits p in [0, inf])

Open
#187,179 1 comment 0 reactions 0 assignees View on GitHub
bot-triaged module: distance functions module: edge cases module: numerical-stability topic: fuzzer triaged
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

### 🐛 Describe the bug

`torch.cdist(x1, x2, p)` evaluates the p-norm directly as `((|x1 - x2|)^p).sum(-1)^(1/p)`. For small `p` on unit-scale input each `(|diff|)^p ~= 1`, so the per-pair sum is `~= M` (the feature dim) and the outer `M^(1/p)` overflows fp32 to `+inf` once `(1/p) * log(M)` exceeds `log(3.4e38) ~= 88.7`. There is no validation on `p`, and the docstring explicitly admits the full range `p in [0, inf]`.

The threshold is much lower than absurd values. For `M = 4` overflow begins at `p ~= 0.027`; for `M = 768` (BERT-scale embeddings) at `p ~= 0.043`. A fractional-norm sweep like `p in {0.01, 0.1, 1, 2}` silently returns all `+inf` for the small-`p` end.

```python
import torch

torch.manual_seed(0)
x1 = torch.rand([2, 3, 4])
x2 = torch.rand([2, 5, 4])
for p in [0.0001, 0.001, 0.01, 0.1, 1.0]:
out = torch.cdist(x1, x2, p=p)
print(f"p={p}: finite_frac={torch.isfinite(out).float().mean().item()}")
```

Observed:

```
p=0.0001: finite_frac=0.0
p=0.001: finite_frac=0.0
p=0.01: finite_frac=0.0
p=0.1: finite_frac=1.0
p=1.0: finite_frac=1.0
```

Expected:

Either (a) raise `ValueError` (or warn) when `(1/p) * log(M)` would exceed `log(dtype_max)`, or (b) evaluate the p-norm in log space via `exp((1/p) * logsumexp(p * log(|diff| + eps)))`, finite for any `p > 0`. Same overflow family as `torch.nn.PairwiseDistance` (#184036, large `p`) and `torch.nn.functional.triplet_margin_loss` (small `p`).

### Versions

```
PyTorch version: 2.5.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.3 LTS (x86_64)
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version: 18.1.3 (1ubuntu1)
CMake version: version 3.28.3
Libc version: glibc-2.39

Python version: 3.10.20 (main, Mar 11 2026, 17:46:40) [GCC 14.3.0] (64-bit runtime)
Python platform: Linux-6.8.0-124-generic-x86_64-with-glibc2.39
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 SUPER
Nvidia driver version: 535.309.01
cuDNN version: Could not collect
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU(s): 20
Model name: 12th Gen Intel(R) Core(TM) i7-12700
Thread(s) per core: 2
Core(s) per socket: 12

Versions of relevant libraries:
[pip3] numpy==2.2.6
[pip3] torch==2.5.1+cu121
[pip3] triton==3.1.0
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.