dotnet / dotnet/spark

[FEATURE REQUEST]: Add support for MapTypes in Vector UDFs

Open
#286 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

There are times when the data stored in the DataFrame's column is a bit more complex, and not just a simple primitive type. Such is the case for maps, arrays and structs. Here is an example of what the schema might look like:
```
root
|-- SomeInteger: integer (nullable = true)
|-- myMap: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
```
When you have a DataFrame like this, you might still end up requiring to perform Vector UDFs over grouped rows, so you would probably would like to do something like this:
```c#
var myDataFrame = otherDataFrame
.GroupBy(otherDataFrame["SomeInteger"])
.Apply(
new StructType(new[]
{
new StructField("SomeInteger", new IntegerType()),
new StructField("myMap", new MapType(new StringType(), new StringType()))
}),
r => SomeVectorUDF(r)
);
```
When running this code, you get the following error `java.lang.UnsupportedOperationException: Unsupported data type: map`. Currently in order to work around this, I require to perform one more UDF first that would serialize the Map into a string, which would be supported by Vector UDFs, and then deserialize it from the UDF side which is less than ideal. Would it be possible to add support for more complex types like Maps for Vector UDFs? cc: @eerhardt

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.