[VL] Spark 4.1: Support NullType Parquet read/write (SPARK-54220)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
## Backend
VL (Velox)
**Gluten version**: main branch
## Description
Spark 4.1 added support for reading and writing NullType columns in Parquet (SPARK-54220). Gluten/Velox does not handle NullType in Parquet format.
Spark 4.1 only.
**Parent issue**: #11910 (`[VL] Spark 4.x: Tracking new feature support`)
### Impact
| Suite | Exclude | spark40 | spark41 |
|-------|---------|:-------:|:-------:|
| GlutenParquetIOSuite | SPARK-54220 | 🟢 | 🔴 |
Note: GlutenParquetIOSuite also has struct field excludes tracked under a separate #11550 sub-issue.
### References
- Apache Spark JIRA: [SPARK-54220](https://issues.apache.org/jira/browse/SPARK-54220)
Contributor guide
Research direction
Start by running GlutenParquetIOSuite for Spark 4.1 and inspect the SPARK-54220 exclusion. Read the linked Apache Spark JIRA and the existing Velox Parquet handling before tracing how NullType is represented. Done means Spark 4.1 Parquet read and write coverage passes without the SPARK-54220 exclusion.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100