[R][C++] Scalar UDFs don't actually deal with scalars
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Noted while testing out UDFs in R. I was wrapping a `system()` call in a UDF to shell out and capture the stdout for each value in the data, but I ended up getting the same result for all rows. After some exploration, I figured out that the problem was that the data going into the UDF is actually a vector, so unless the R UDF function is properly vectorized, you'll get unexpected data.
Here's an example that illustrates:
```Java
register_scalar_function(
"test",
function(context, x) paste(x, collapse=","),
utf8(),
utf8(),
auto_convert=TRUE
)
Table$create(x = c("a", "b", "c")) |>
transmute(test(x)) |>
collect()
# # A tibble: 3 × 1
# `test(x)`
#
# 1 a,b,c
# 2 a,b,c
# 3 a,b,c
```
Basically, the UDF gets the chunk of data and evaluates to return a Scalar, which gets recycled for all rows.
**Reporter**: [Neal Richardson](https://issues.apache.org/jira/browse/ARROW-17437) / @nealrichardson
**Note**: *This issue was originally created as [ARROW-17437](https://issues.apache.org/jira/browse/ARROW-17437). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the R example using register_scalar_function and a table with multiple values, then trace how the UDF receives the data chunk and returns a Scalar. Done means a scalar UDF evaluates per input value rather than returning one chunk result that is recycled for every row.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, r
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100