apache / apache/arrow-rs

Support RunEndEncoded arrays in arrow-arith numeric kernels (add/sub/mul/div)

Open
#10,891 2 comments 0 reactions 1 assignee Claimed by @yinli-systems View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Is your feature request related to a problem or challenge?

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

`arrow-arith`'s elementwise numeric kernels don't handle `DataType::RunEndEncoded`. The type dispatch lives in [`arithmetic_op`](https://github.com/apache/arrow-rs/blob/58.4.0/arrow-arith/src/numeric.rs#L220), called from the public [`add`](https://github.com/apache/arrow-rs/blob/58.4.0/arrow-arith/src/numeric.rs#L34)/[`add_wrapping`](https://github.com/apache/arrow-rs/blob/58.4.0/arrow-arith/src/numeric.rs#L39)/[`sub`](https://github.com/apache/arrow-rs/blob/58.4.0/arrow-arith/src/numeric.rs#L44)/[`sub_wrapping`](https://github.com/apache/arrow-rs/blob/58.4.0/arrow-arith/src/numeric.rs#L49) entry points. It has no `RunEndEncoded` match arm, so a `RunEndEncoded`-wrapped operand falls through to the generic [fallback error arm](https://github.com/apache/arrow-rs/blob/58.4.0/arrow-arith/src/numeric.rs#L259) and returns `ArrowError::InvalidArgumentError`, e.g.:

Invalid arithmetic operation: RunEndEncoded(...) + Int64

We hit this in [Apache DataFusion](https://github.com/apache/datafusion/pull/24565): a `RANGE PRECEDING/FOLLOWING` window frame ordered by an REE-encoded column needs to add/subtract the frame offset from each row's ordering value.

**Describe the solution you'd like**

Add `RunEndEncoded` support to arithmetic_op

**Describe alternatives you've considered**

- Decode REE to a flat array before arithmetic: works today as a caller-side workaround, but throws away REE's compression benefit for exactly the workloads (e.g. windowed aggregates over repetitive columns) that make REE worth using in the first place.
- Leave callers to detect and reject REE before reaching arithmetic: what DataFusion does today (see linked PR) — safe, but means REE columns can't be used in RANGE window frames or similar arithmetic-dependent operations at all until this lands.

**Additional context**

Related prior art: #3520 (REE epic — still has an open "Support REE in compute kernels" checklist item), #9620 (REE support for `arrow-ord` comparison kernels, landed).

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.