apache / apache/datafusion

FFI Scalar function wrapper does not preserve Scalar/Array-ness of a ColumnarValue

Open
#18,201 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

In the FFI_ScalarUDF, I noticed that the Scalar/Array nature of an argument is not preserved. This is not a bug, as scalar values are correctly repeated to the number of rows, but it does make the FFI interface of limited value for what I'd like to do with it (because some geometry predicates are significantly faster if we "prepare" a scalar value once per batch instead of once per row).

### Describe the solution you'd like

I think we can use a comparison between the `num_rows` argument and length of the wrapper array to decide if an argument is a scalar or an array (with a corner case when there's `num_rows` of 1). This would break the FFI on the "new function producer but old function caller" end although I don't think it would segfault anything if it's of interest.

### Describe alternatives you've considered

Our own FFI (or just building everything at once)

### Additional context

https://github.com/apache/datafusion/blob/b5b7f9b356a5363a71ee1d293df199c9e33206cd/datafusion/ffi/src/udf/mod.rs#L185-L191

https://github.com/apache/datafusion/blob/b5b7f9b356a5363a71ee1d293df199c9e33206cd/datafusion/ffi/src/udf/mod.rs#L389-L391

PR where I discovered this: https://github.com/apache/sedona-db/pull/228

cc @kylebarron who may run across this issue

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/ffi/src/udf/mod.rs at the referenced wrapper construction and argument handling around lines 185-191 and 389-391. Trace how num_rows and wrapper-array length currently determine argument representation, including the one-row corner case and producer/caller compatibility. Done means the FFI scalar UDF preserves whether each argument was scalar or array without breaking the intended row expansion behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.