dotnet / dotnet/spark

[FEATURE REQUEST]: Support variable length inputs to UDF

Open
#200 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

Coming from pySpark, one of the handy features when writing Pandas UDF is the ability to pass a variable list of input columns. It would seem in the current implementation of Spark .NET that you can only provide fixed length argument lists.

For example, consider this Python code:

```python
def generate_pandas_udf(colnames):
@pandas_udf('double')
def formulaFunc(*columns):
# column names are accessed in colnames
# column vectors are unpacked from columns
summed = 0
for c in cols:
summed += c
return summed

columns_to_gather = ['col1','col2','col3']
df.withColumn('col4',generate_pandas_udf(columns_to_gather)(*columns_to_gather)))
```

I can easily write a generic UDF function that accepts a variable list of columns like the above, there is no need for me to know in advance how many arguments i wish to pass.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.