Parquet writer performance degrades with deeply nested all-null columns
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## Problem
The Parquet writer performs substantial processing for deeply nested columns even when entire nested subtrees are null and contain no values.
The cost grows with schema depth and number of leaf columns rather than with the amount of data that ultimately needs to be encoded. This particularly affects schema-evolution workloads, where newly introduced nested fields are commonly all null.
## Observed behavior
For all-null nested leaves, the writer still performs work across several stages:
- Constructing internal Parquet column representations
- Generating Dremel definition and repetition levels
- Scanning values and nested storage to calculate fragment sizes
- Recursively estimating the sizes of LIST and STRUCT children
- Preparing and encoding pages through the normal GPU path
Much of this processing produces no encoded values because the affected leaves are entirely null.
In the captured workload, the Nsight Systems trace shows the `GPU file format write` range taking **32.146 seconds** within a **53.624-second** Spark task. The nested `write` range alone takes **20.635 seconds**.
Contributor guide
Research direction
No source file or test entry point is named. Start by reproducing a deeply nested workload with all-null leaves and inspect the Nsight Systems trace around the GPU file format write and nested write ranges. Trace the Parquet writer stages described in the issue; done means avoiding unnecessary processing and encoding for entirely null nested subtrees while preserving correct output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100