apache / apache/gluten

[VL] Spark 4.x: Support Variant type

Open
#11,371 6 comments 0 reactions 1 assignee Claimed by @baibaichen View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

Spark-4.0 has added new data type `VariantType`. The support for VariantType and variant shredded data has been added to parquet reader and writer.

Refer:
- https://issues.apache.org/jira/browse/SPARK-50614
- [[SPARK-48495](https://issues.apache.org/jira/browse/SPARK-48495)][SQL][DOCS] Describe shredding scheme for Variant
- [[SPARK-50614](https://issues.apache.org/jira/browse/SPARK-50614)][SQL] Add Variant shredding support for Parquet
- [[SPARK-54454](https://issues.apache.org/jira/browse/SPARK-54454)][SQL] Enable variant shredding and variant logical type annotation configs by default

This feature has been enabled by default in Spark. Currently, since Gluten does not support `VariantType`, need to add fallback.

### Gluten version

None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.