NVIDIA / NVIDIA/cudf

[PERF]: Reuse parquet metadata from polars

Open
#22,666 0 comments 0 reactions 0 assignees View on GitHub
cudf-polars cuIO feature request Performance
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

cudf-polars currently reads parquet metadata in a few spots. Most notably,

- `Scan.do_evaluate` with the `ChunkedParquetReader`, inside the `reader.read_chunk()`: https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/dsl/ir.py#L829-L834
- `Scan.do_evaluate` with the non-chunked, regular `plc.io.parquet.read_parquet`: https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/dsl/ir.py#L864-L866
- Building a `filter` expression for the rapidsmpf reader: https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/streaming/actor_graph/io.py#L580-L585
- As part of `SplitScan.do_evaluate`: https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/streaming/io.py#L209-L211, to generate the `Scan` nodes dynamically
- As part of `ParquetMetadata.__init__` in stats collection (https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/streaming/io.py#L515-L517)

We occasionally *re*read parquet metadata that polars has already read and parsed. https://github.com/pola-rs/polars/issues/27756 is a feature request to polars to expose some of the information it already has, which we can use when translating a `Scan` node. We can avoid rereading / reparsing the data.

One caveat: polars generally just reads a single file per table. A `scan_parquet` node might address many `paths`, but polars doesn't want to read the metadata of *every* path. But for single-file tables, we might be able to avoid rereading metadata entirely.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.