[Python] Allow calling UDF kernels with field/scalar expressions
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
From https://github.com/apache/arrow/pull/13687#issuecomment-1240399112, where it came up while adding documentation on how to use UDFs in Python. When just wanting to invoke a UDF with arrays, you can do `pc.call_function("my_udf", [pc.field("a")])`.
But if you want to use your UDF in a context that needs an expression (eg a dataset projection), you need to be able to call the UDF with expressions as argument. And currently, the `pc.call_function` doesn't work that way (it expects actual, materialized arrays/scalars as arguments). As a workaround, you can use the private `Expression._call`:
```python
# doesn't work with expressions
>>> pc.call_function("my_udf", [pc.field("col")])
...
TypeError: Got unexpected argument type for compute function
# workaround
>>> pc.Expression._call("my_udf", [pc.field("col")])
```
So we should try to improve the usability here. Some options:
- See if we can change `pc.call_function` to also accept Expressions as arguments
- Make the `_call` public, so one can do `pc.Expression.call("my_udf", [..])`
cc @westonpace @vibhatha
**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-17827) / @jorisvandenbossche
**Note**: *This issue was originally created as [ARROW-17827](https://issues.apache.org/jira/browse/ARROW-17827). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by comparing the public pc.call_function entry point with the private pc.Expression._call workaround shown in the issue. Check how each handles field and scalar expressions, including use in a dataset projection, and determine which proposed interface should be supported. Done means a documented public way to invoke UDFs with expressions, with the relevant behavior verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100