apache / apache/arrow

[R][C++] Scalar UDFs don't actually deal with scalars

Open
#32,704 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Noted while testing out UDFs in R. I was wrapping a `system()` call in a UDF to shell out and capture the stdout for each value in the data, but I ended up getting the same result for all rows. After some exploration, I figured out that the problem was that the data going into the UDF is actually a vector, so unless the R UDF function is properly vectorized, you'll get unexpected data.

Here's an example that illustrates:

```Java

register_scalar_function(
"test",
function(context, x) paste(x, collapse=","),
utf8(),
utf8(),
auto_convert=TRUE
)

Table$create(x = c("a", "b", "c")) |>
transmute(test(x)) |>
collect()

# # A tibble: 3 × 1
# `test(x)`
#
# 1 a,b,c
# 2 a,b,c
# 3 a,b,c
```

Basically, the UDF gets the chunk of data and evaluates to return a Scalar, which gets recycled for all rows.

**Reporter**: [Neal Richardson](https://issues.apache.org/jira/browse/ARROW-17437) / @nealrichardson

**Note**: *This issue was originally created as [ARROW-17437](https://issues.apache.org/jira/browse/ARROW-17437). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the R example using register_scalar_function and a table with multiple values, then trace how the UDF receives the data chunk and returns a Scalar. Done means a scalar UDF evaluates per input value rather than returning one chunk result that is recycled for every row.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, r
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.