apache / apache/gluten

[VL] Spark 4.1: Support NullType Parquet read/write (SPARK-54220)

Open
#11,921 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

## Backend
VL (Velox)

**Gluten version**: main branch

## Description

Spark 4.1 added support for reading and writing NullType columns in Parquet (SPARK-54220). Gluten/Velox does not handle NullType in Parquet format.

Spark 4.1 only.

**Parent issue**: #11910 (`[VL] Spark 4.x: Tracking new feature support`)

### Impact

| Suite | Exclude | spark40 | spark41 |
|-------|---------|:-------:|:-------:|
| GlutenParquetIOSuite | SPARK-54220 | 🟢 | 🔴 |

Note: GlutenParquetIOSuite also has struct field excludes tracked under a separate #11550 sub-issue.

### References

- Apache Spark JIRA: [SPARK-54220](https://issues.apache.org/jira/browse/SPARK-54220)

Contributor guide

Open the contributing guide

Research direction

Start by running GlutenParquetIOSuite for Spark 4.1 and inspect the SPARK-54220 exclusion. Read the linked Apache Spark JIRA and the existing Velox Parquet handling before tracing how NullType is represented. Done means Spark 4.1 Parquet read and write coverage passes without the SPARK-54220 exclusion.

Written by the indexing model from the issue text.

Assessment

Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.