dotnet / dotnet/spark

[FEATURE REQUEST]: Consider adding support for ValueTuples when defining UDFs

Open
#285 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

It is not uncommon to have people use DataFrames in Scala with schemas with structs as the following example show:
```
root
|-- _1: struct (nullable = true)
| |-- _1: integer (nullable = false)
| |-- _2: string (nullable = true)
|-- _2: integer (nullable = false)
```
When developing a a custom UDF in Spark.NET that would take a int and string, it would be very handy to be able to do something like the following:
```c#
Func CreateStructColumn = Udf((someInt, someString) => (someInt.Value, someString));
```

Currently if we do this, we get an exception message saying that ValueTuple is not supported. Is there a different way to do this currently? If not, could we add support for using ValueTuples as a representation of scala structs?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.