[VL] Spark 4.x: Support Variant type
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
Spark-4.0 has added new data type `VariantType`. The support for VariantType and variant shredded data has been added to parquet reader and writer.
Refer:
- https://issues.apache.org/jira/browse/SPARK-50614
- [[SPARK-48495](https://issues.apache.org/jira/browse/SPARK-48495)][SQL][DOCS] Describe shredding scheme for Variant
- [[SPARK-50614](https://issues.apache.org/jira/browse/SPARK-50614)][SQL] Add Variant shredding support for Parquet
- [[SPARK-54454](https://issues.apache.org/jira/browse/SPARK-54454)][SQL] Enable variant shredding and variant logical type annotation configs by default
This feature has been enabled by default in Spark. Currently, since Gluten does not support `VariantType`, need to add fallback.
### Gluten version
None
Contributor guide
Assessment
This issue has not been assessed yet.