FFI Scalar function wrapper does not preserve Scalar/Array-ness of a ColumnarValue
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
In the FFI_ScalarUDF, I noticed that the Scalar/Array nature of an argument is not preserved. This is not a bug, as scalar values are correctly repeated to the number of rows, but it does make the FFI interface of limited value for what I'd like to do with it (because some geometry predicates are significantly faster if we "prepare" a scalar value once per batch instead of once per row).
### Describe the solution you'd like
I think we can use a comparison between the `num_rows` argument and length of the wrapper array to decide if an argument is a scalar or an array (with a corner case when there's `num_rows` of 1). This would break the FFI on the "new function producer but old function caller" end although I don't think it would segfault anything if it's of interest.
### Describe alternatives you've considered
Our own FFI (or just building everything at once)
### Additional context
https://github.com/apache/datafusion/blob/b5b7f9b356a5363a71ee1d293df199c9e33206cd/datafusion/ffi/src/udf/mod.rs#L185-L191
https://github.com/apache/datafusion/blob/b5b7f9b356a5363a71ee1d293df199c9e33206cd/datafusion/ffi/src/udf/mod.rs#L389-L391
PR where I discovered this: https://github.com/apache/sedona-db/pull/228
cc @kylebarron who may run across this issue
Contributor guide
Research direction
Start in datafusion/ffi/src/udf/mod.rs at the referenced wrapper construction and argument handling around lines 185-191 and 389-391. Trace how num_rows and wrapper-array length currently determine argument representation, including the one-row corner case and producer/caller compatibility. Done means the FFI scalar UDF preserves whether each argument was scalar or array without breaking the intended row expansion behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100