apache / apache/gluten

[VL] Spark 4.1: Support Parquet struct field compatibility improvements

Open
#11,914 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

## Backend
VL (Velox)

## Bug description

Two Parquet struct field handling issues in GlutenParquetIOSuite on Spark 4.1:

**Issue 1: SPARK-53535** — Spark 4.1 added `returnNullStructIfAllFieldsMissing` parameter for missing struct field handling. Gluten has not adapted to this change.

**Issue 2: Vectorized reader missing struct fields** — Pre-existing test (SPARK-34863) for vectorized reader complex type handling. SPARK-53535 modified this test, causing it to fail on Spark 4.1 with Gluten.

These are two independent root causes but both only affect Spark 4.1.

Parent issue: #11550

### Impact

| Suite | Exclude | spark40 | spark41 |
|-------|---------|:-------:|:-------:|
| GlutenParquetIOSuite | SPARK-53535 | 🟢 | 🔴 |
| GlutenParquetIOSuite | vectorized reader: missing all struct fields | 🟢 | 🔴 |

Note: GlutenParquetIOSuite also has a SPARK-54220 (NullType) exclude tracked separately under the Tracking issue #11910.

### References

- Apache Spark JIRA: [SPARK-53535](https://issues.apache.org/jira/browse/SPARK-53535)
- Apache Spark JIRA: [SPARK-34863](https://issues.apache.org/jira/browse/SPARK-34863)

Contributor guide

Open the contributing guide

Research direction

Start with GlutenParquetIOSuite and run the Spark 4.1 cases associated with SPARK-53535 and SPARK-34863. Read the linked Spark JIRA references to understand the changed struct-field behavior, then trace the two independent failures in the suite. Done means both Spark 4.1 cases pass without the reported exclusions while Spark 4.0 remains passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.