Add vector distance, array math, and array aggregate functions
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Summary
This issue tracks adding vector math and array aggregate scalar functions to DataFusion. These close gaps versus DuckDB and LanceDB for vector search and array analytics workloads.
Replaces #21371 and #21376, which were requested to be split into function-per-PR submissions (per @alamb's review).
## Functions
### Vector math (with shared `vector_math.rs` primitives)
| Function | Signature | Reference |
|----------|-----------|-----------|
| `cosine_distance` | `(array, array) → float64` | [DuckDB `array_cosine_similarity`](https://duckdb.org/docs/sql/functions/array.html) |
| `inner_product` | `(array, array) → float64` | [DuckDB `array_inner_product`](https://duckdb.org/docs/sql/functions/array.html) |
| `array_normalize` | `(array) → array` | NumPy / scipy convention |
### Array element-wise math
| Function | Signature | Reference |
|----------|-----------|-----------|
| `array_add` | `(array, array) → array` | Element-wise addition |
| `array_subtract` | `(array, array) → array` | Element-wise subtraction |
| `array_scale` | `(array, scalar) → array` | Scalar multiply |
### Array aggregate scalars
| Function | Signature | Reference |
|----------|-----------|-----------|
| `array_sum` / `list_sum` | `(array) → numeric` | [DuckDB `list_sum`](https://duckdb.org/docs/sql/functions/list.html#list_sumlist) |
| `array_product` / `list_product` | `(array) → numeric` | [DuckDB `list_product`](https://duckdb.org/docs/sql/functions/list.html) |
| `array_avg` / `list_avg` | `(array) → float64` | [DuckDB `list_avg`](https://duckdb.org/docs/sql/functions/list.html) |
### Alias fix
| Fix | Description |
|-----|-------------|
| `list_min` | Missing alias on `ArrayMin` (parity with existing `list_max` on `ArrayMax`) |
## Submission plan
One PR per function, submitted serially. Each PR will reference this issue.
## References
- [DuckDB list functions](https://duckdb.org/docs/sql/functions/list.html)
- [DuckDB array functions](https://duckdb.org/docs/sql/functions/array.html)
- [LanceDB distance metrics](https://lancedb.github.io/lancedb/)
- [Trino array functions](https://trino.io/docs/current/functions/array.html)
Contributor guide
Research direction
The issue names no source files, tests, or entry points. Start by reviewing the existing DataFusion array-function implementations and the linked DuckDB, LanceDB, and Trino references, then determine how the shared vector_math.rs primitives and each function fit the one-PR-per-function plan. Done means the listed functions and the list_min alias are covered by serial submissions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100