NVIDIA / NVIDIA/cudf

[FEA] Parquet chunked reader benchmark that emulates heavily nested column data with specific properties.

Open
#16,130 0 comments 0 reactions 1 assignee Assigned to @pmattione-nvidia View on GitHub
cuIO feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

The chunked reader suffers a bit of a perfect storm of performance issues when it comes to certain shapes of input data.

- Many many columns being skewed towards a small number of the output columns containing almost all of the additional nesting.
- Heavy use of strings and lists.
- Using zstd compression, which dramatically increases the decompression memory overhead (2.5x the total decompressed size needed just for temp space. So in the best case, a 4 GB chunk is only really getting 1.1 GB of usable data to decode, which exaggerates the number of chunks that have to happen).

The customer data case we were seeing where this was causing issues was:

- Hundreds of output columns, containing thousands input columns in a nested way. Much of the nesting was contained in only a few output columns. Strings, lists, and using zstd.
- This resulted in 4 GB batches where we were reading only 20k rows at a time, and in some cases were processing a single page of data for many individual columns..

Optimizing this is a broad subject so having a nice baseline in the cudf benchmarks to start with would be very helpful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.