NVIDIA / NVIDIA/cudf

[FEA] Cache schema tree in `chunked_parquet_writer`

Open
#19,060 3 comments 0 reactions 0 assignees View on GitHub
cuIO feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
When using `cudf::io::chunked_parquet_writer`, the schema tree is recomputed for every chunk passed to `writer.write()`. This introduces unnecessary overhead when writing many chunks with the same schema.

**Describe the solution you'd like**
Cache the schema tree after the first `write()` call and reuse it for subsequent chunks, assuming the schema is identical.

**Describe alternatives you've considered**
Continue with the status-quo. Ie. recomputing the schema tree for every chunk. Still inefficient for workloads with a high number of chunks.

**Additional context**
libcudf chunked write benchmark for reference: https://github.com/rapidsai/cudf/pull/19015#issuecomment-2922368382

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.