[FEATURE REQUEST]: Add support for MapTypes in Vector UDFs
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
There are times when the data stored in the DataFrame's column is a bit more complex, and not just a simple primitive type. Such is the case for maps, arrays and structs. Here is an example of what the schema might look like:
```
root
|-- SomeInteger: integer (nullable = true)
|-- myMap: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
```
When you have a DataFrame like this, you might still end up requiring to perform Vector UDFs over grouped rows, so you would probably would like to do something like this:
```c#
var myDataFrame = otherDataFrame
.GroupBy(otherDataFrame["SomeInteger"])
.Apply(
new StructType(new[]
{
new StructField("SomeInteger", new IntegerType()),
new StructField("myMap", new MapType(new StringType(), new StringType()))
}),
r => SomeVectorUDF(r)
);
```
When running this code, you get the following error `java.lang.UnsupportedOperationException: Unsupported data type: map`. Currently in order to work around this, I require to perform one more UDF first that would serialize the Map into a string, which would be supported by Vector UDFs, and then deserialize it from the UDF side which is less than ideal. Would it be possible to add support for more complex types like Maps for Vector UDFs? cc: @eerhardt
Contributor guide
Assessment
This issue has not been assessed yet.