apache / apache/gluten

[VL] Parquet writer marks all struct fields as OPTIONAL, breaking Spark variant type

Open
#11,803 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

Description

Velox parquet writer unconditionally sets all struct fields as OPTIONAL (nullable), even when Spark schema specifies REQUIRED. This breaks Spark's variant binary format validation.

```
enableSuite[GlutenVariantSuite]
// TODO: Velox parquet writer marks all struct fields as OPTIONAL (nullable),
// but Spark's variant type requires REQUIRED fields. Needs Velox-side fix.
.exclude("SPARK-47546: invalid variant binary")
.exclude("SPARK-47546: valid variant binary")
```

Root cause:

RowType has no nullability → exportToArrow() sets ARROW_FLAG_NULLABLE on all fields → Arrow Parquet Writer writes OPTIONAL → Spark convertVariantField expects REQUIRED → error.

Affected tests

SPARK-47546: valid variant binary
SPARK-47546: invalid variant binary

### Gluten version

_No response_

### Spark version

4.0, 4.1

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by tracing RowType through exportToArrow() and the Arrow Parquet Writer, focusing on how ARROW_FLAG_NULLABLE is assigned. Use the affected tests, “SPARK-47546: valid variant binary” and “SPARK-47546: invalid variant binary,” to verify that required struct fields remain required and Spark variant validation passes.

Written by the indexing model from the issue text.

Assessment

Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.