apache / apache/datafusion

Use Specialization Instead of ScalarValue Binary Operations

Open
#6,842 2 comments 3 reactions 1 assignee Claimed by @tustvold View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Currently a number of operations are implemented directly on ScalarValue, including:

* Arithmetic
* Logical And
* Logical Or
* BitAnd, BitXor, etc...
* Comparison

Not only does this result in a huge amount of code, but also these operations don't behave the same way as their array counterparts.

For example:

* An operation on a null doesn't yield a null
* Floating point NaNs are not total ordered
* No support for decimals
* Differing support for interval arithmetic compared to the array kernels

### Describe the solution you'd like

These kernels largely appear to exist for the purposes of aggregation, where the aggregated types are known statically. We should replace these uses with specialization, as done in https://github.com/apache/arrow-datafusion/pull/6800#discussion_r1248104156. The remaining uses should make use of the new Datum abstraction https://github.com/apache/arrow-rs/pull/4393 to use the same arrow-rs kernels https://github.com/apache/arrow-rs/pull/4465

### Describe alternatives you've considered

_No response_

### Additional context

#4973 tracks improving the aggregator performance
https://github.com/apache/arrow-datafusion/pull/6832 updates DF to use the Datum kernels

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.