apache / apache/datafusion

Add vector distance, array math, and array aggregate functions

Open
#21,536 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Summary

This issue tracks adding vector math and array aggregate scalar functions to DataFusion. These close gaps versus DuckDB and LanceDB for vector search and array analytics workloads.

Replaces #21371 and #21376, which were requested to be split into function-per-PR submissions (per @alamb's review).

## Functions

### Vector math (with shared `vector_math.rs` primitives)

| Function | Signature | Reference |
|----------|-----------|-----------|
| `cosine_distance` | `(array, array) → float64` | [DuckDB `array_cosine_similarity`](https://duckdb.org/docs/sql/functions/array.html) |
| `inner_product` | `(array, array) → float64` | [DuckDB `array_inner_product`](https://duckdb.org/docs/sql/functions/array.html) |
| `array_normalize` | `(array) → array` | NumPy / scipy convention |

### Array element-wise math

| Function | Signature | Reference |
|----------|-----------|-----------|
| `array_add` | `(array, array) → array` | Element-wise addition |
| `array_subtract` | `(array, array) → array` | Element-wise subtraction |
| `array_scale` | `(array, scalar) → array` | Scalar multiply |

### Array aggregate scalars

| Function | Signature | Reference |
|----------|-----------|-----------|
| `array_sum` / `list_sum` | `(array) → numeric` | [DuckDB `list_sum`](https://duckdb.org/docs/sql/functions/list.html#list_sumlist) |
| `array_product` / `list_product` | `(array) → numeric` | [DuckDB `list_product`](https://duckdb.org/docs/sql/functions/list.html) |
| `array_avg` / `list_avg` | `(array) → float64` | [DuckDB `list_avg`](https://duckdb.org/docs/sql/functions/list.html) |

### Alias fix

| Fix | Description |
|-----|-------------|
| `list_min` | Missing alias on `ArrayMin` (parity with existing `list_max` on `ArrayMax`) |

## Submission plan

One PR per function, submitted serially. Each PR will reference this issue.

## References

- [DuckDB list functions](https://duckdb.org/docs/sql/functions/list.html)
- [DuckDB array functions](https://duckdb.org/docs/sql/functions/array.html)
- [LanceDB distance metrics](https://lancedb.github.io/lancedb/)
- [Trino array functions](https://trino.io/docs/current/functions/array.html)

Contributor guide

Open the contributing guide

Research direction

The issue names no source files, tests, or entry points. Start by reviewing the existing DataFusion array-function implementations and the linked DuckDB, LanceDB, and Trino references, then determine how the shared vector_math.rs primitives and each function fit the one-PR-per-function plan. Done means the listed functions and the list_min alias are covered by serial submissions.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.