[VL] Parquet writer marks all struct fields as OPTIONAL, breaking Spark variant type
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
Description
Velox parquet writer unconditionally sets all struct fields as OPTIONAL (nullable), even when Spark schema specifies REQUIRED. This breaks Spark's variant binary format validation.
```
enableSuite[GlutenVariantSuite]
// TODO: Velox parquet writer marks all struct fields as OPTIONAL (nullable),
// but Spark's variant type requires REQUIRED fields. Needs Velox-side fix.
.exclude("SPARK-47546: invalid variant binary")
.exclude("SPARK-47546: valid variant binary")
```
Root cause:
RowType has no nullability → exportToArrow() sets ARROW_FLAG_NULLABLE on all fields → Arrow Parquet Writer writes OPTIONAL → Spark convertVariantField expects REQUIRED → error.
Affected tests
SPARK-47546: valid variant binary
SPARK-47546: invalid variant binary
### Gluten version
_No response_
### Spark version
4.0, 4.1
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by tracing RowType through exportToArrow() and the Arrow Parquet Writer, focusing on how ARROW_FLAG_NULLABLE is assigned. Use the affected tests, “SPARK-47546: valid variant binary” and “SPARK-47546: invalid variant binary,” to verify that required struct fields remain required and Spark variant validation passes.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100