[FEA] Parquet chunked reader benchmark that emulates heavily nested column data with specific properties.
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
The chunked reader suffers a bit of a perfect storm of performance issues when it comes to certain shapes of input data.
- Many many columns being skewed towards a small number of the output columns containing almost all of the additional nesting.
- Heavy use of strings and lists.
- Using zstd compression, which dramatically increases the decompression memory overhead (2.5x the total decompressed size needed just for temp space. So in the best case, a 4 GB chunk is only really getting 1.1 GB of usable data to decode, which exaggerates the number of chunks that have to happen).
The customer data case we were seeing where this was causing issues was:
- Hundreds of output columns, containing thousands input columns in a nested way. Much of the nesting was contained in only a few output columns. Strings, lists, and using zstd.
- This resulted in 4 GB batches where we were reading only 20k rows at a time, and in some cases were processing a single page of data for many individual columns..
Optimizing this is a broad subject so having a nice baseline in the cudf benchmarks to start with would be very helpful.
Contributor guide
Assessment
This issue has not been assessed yet.