apache / apache/datafusion

Vector functions overflow or underflow for finite Float64 inputs

Open
#25,265 2 comments 0 reactions 1 assignee Claimed by @namanjain24-sudo View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I found that `array_distance`, `cosine_distance`, and `array_normalize` directly sum squared `Float64` values. The intermediate squares can overflow or underflow even when the correct result is representable.

This can turn nonzero distances into zero, identical vectors into `NaN`, and valid normalized vectors into zero vectors.

### To Reproduce

Tested on 55.1.0 and `main` at `9082d6b10`.

```sql
SELECT array_distance([CAST(1e-200 AS DOUBLE)], [CAST(0 AS DOUBLE)]);
-- actual: 0.0; expected: 1e-200

SELECT cosine_distance(
[CAST(3e200 AS DOUBLE), CAST(4e200 AS DOUBLE)],
[CAST(3e200 AS DOUBLE), CAST(4e200 AS DOUBLE)]
);
-- actual: NaN; expected: 0.0

SELECT array_normalize([CAST(3e200 AS DOUBLE), CAST(4e200 AS DOUBLE)]);
-- actual: [0.0, 0.0]; expected: [0.6, 0.8]
```

### Expected behavior

Finite inputs should not produce zero, `NaN`, or a zero vector when the mathematical result is finite and representable.

### Additional context

The same unscaled sum-of-squares pattern appears in [`array_distance`](https://github.com/apache/datafusion/blob/9082d6b10c29b72d56bede3d8e353d9d61fde542/datafusion/functions-nested/src/distance.rs#L192-L201), [`cosine_distance`](https://github.com/apache/datafusion/blob/9082d6b10c29b72d56bede3d8e353d9d61fde542/datafusion/functions-nested/src/cosine_distance.rs#L200-L214), and [`array_normalize`](https://github.com/apache/datafusion/blob/9082d6b10c29b72d56bede3d8e353d9d61fde542/datafusion/functions-nested/src/array_normalize.rs#L179-L195). I couldn't find an existing issue for this behavior.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.