[FEATURE REQUEST]: Support variable length inputs to UDF
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
Coming from pySpark, one of the handy features when writing Pandas UDF is the ability to pass a variable list of input columns. It would seem in the current implementation of Spark .NET that you can only provide fixed length argument lists.
For example, consider this Python code:
```python
def generate_pandas_udf(colnames):
@pandas_udf('double')
def formulaFunc(*columns):
# column names are accessed in colnames
# column vectors are unpacked from columns
summed = 0
for c in cols:
summed += c
return summed
columns_to_gather = ['col1','col2','col3']
df.withColumn('col4',generate_pandas_udf(columns_to_gather)(*columns_to_gather)))
```
I can easily write a generic UDF function that accepts a variable list of columns like the above, there is no need for me to know in advance how many arguments i wish to pass.
Contributor guide
Assessment
This issue has not been assessed yet.