[VL] Spark 4.1: Support Parquet struct field compatibility improvements
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
## Backend
VL (Velox)
## Bug description
Two Parquet struct field handling issues in GlutenParquetIOSuite on Spark 4.1:
**Issue 1: SPARK-53535** — Spark 4.1 added `returnNullStructIfAllFieldsMissing` parameter for missing struct field handling. Gluten has not adapted to this change.
**Issue 2: Vectorized reader missing struct fields** — Pre-existing test (SPARK-34863) for vectorized reader complex type handling. SPARK-53535 modified this test, causing it to fail on Spark 4.1 with Gluten.
These are two independent root causes but both only affect Spark 4.1.
Parent issue: #11550
### Impact
| Suite | Exclude | spark40 | spark41 |
|-------|---------|:-------:|:-------:|
| GlutenParquetIOSuite | SPARK-53535 | 🟢 | 🔴 |
| GlutenParquetIOSuite | vectorized reader: missing all struct fields | 🟢 | 🔴 |
Note: GlutenParquetIOSuite also has a SPARK-54220 (NullType) exclude tracked separately under the Tracking issue #11910.
### References
- Apache Spark JIRA: [SPARK-53535](https://issues.apache.org/jira/browse/SPARK-53535)
- Apache Spark JIRA: [SPARK-34863](https://issues.apache.org/jira/browse/SPARK-34863)
Contributor guide
Research direction
Start with GlutenParquetIOSuite and run the Spark 4.1 cases associated with SPARK-53535 and SPARK-34863. Read the linked Spark JIRA references to understand the changed struct-field behavior, then trace the two independent failures in the suite. Done means both Spark 4.1 cases pass without the reported exclusions while Spark 4.0 remains passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100