NVIDIA / NVIDIA/cudf

Parquet writer performance degrades with deeply nested all-null columns

Open
#23,465 1 comment 0 reactions 0 assignees View on GitHub
cuIO
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Problem

The Parquet writer performs substantial processing for deeply nested columns even when entire nested subtrees are null and contain no values.

The cost grows with schema depth and number of leaf columns rather than with the amount of data that ultimately needs to be encoded. This particularly affects schema-evolution workloads, where newly introduced nested fields are commonly all null.

## Observed behavior

For all-null nested leaves, the writer still performs work across several stages:

- Constructing internal Parquet column representations
- Generating Dremel definition and repetition levels
- Scanning values and nested storage to calculate fragment sizes
- Recursively estimating the sizes of LIST and STRUCT children
- Preparing and encoding pages through the normal GPU path

Much of this processing produces no encoded values because the affected leaves are entirely null.

In the captured workload, the Nsight Systems trace shows the `GPU file format write` range taking **32.146 seconds** within a **53.624-second** Spark task. The nested `write` range alone takes **20.635 seconds**.

Image

Contributor guide

Open the contributing guide

Research direction

No source file or test entry point is named. Start by reproducing a deeply nested workload with all-null leaves and inspect the Nsight Systems trace around the GPU file format write and nested write ranges. Trace the Parquet writer stages described in the issue; done means avoiding unnecessary processing and encoding for entirely null nested subtrees while preserving correct output.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.