apache / apache/arrow

[Python] Allow calling UDF kernels with field/scalar expressions

Open
#33,048 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

From https://github.com/apache/arrow/pull/13687#issuecomment-1240399112, where it came up while adding documentation on how to use UDFs in Python. When just wanting to invoke a UDF with arrays, you can do `pc.call_function("my_udf", [pc.field("a")])`.

But if you want to use your UDF in a context that needs an expression (eg a dataset projection), you need to be able to call the UDF with expressions as argument. And currently, the `pc.call_function` doesn't work that way (it expects actual, materialized arrays/scalars as arguments). As a workaround, you can use the private `Expression._call`:

```python

# doesn't work with expressions
>>> pc.call_function("my_udf", [pc.field("col")])
...
TypeError: Got unexpected argument type for compute function
# workaround
>>> pc.Expression._call("my_udf", [pc.field("col")])

```

So we should try to improve the usability here. Some options:

- See if we can change `pc.call_function` to also accept Expressions as arguments
- Make the `_call` public, so one can do `pc.Expression.call("my_udf", [..])`

cc @westonpace @vibhatha

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-17827) / @jorisvandenbossche

**Note**: *This issue was originally created as [ARROW-17827](https://issues.apache.org/jira/browse/ARROW-17827). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by comparing the public pc.call_function entry point with the private pc.Expression._call workaround shown in the issue. Check how each handles field and scalar expressions, including use in a dataset projection, and determine which proposed interface should be supported. Done means a documented public way to invoke UDFs with expressions, with the relevant behavior verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.