[PERF]: Reuse parquet metadata from polars
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
cudf-polars currently reads parquet metadata in a few spots. Most notably,
- `Scan.do_evaluate` with the `ChunkedParquetReader`, inside the `reader.read_chunk()`: https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/dsl/ir.py#L829-L834
- `Scan.do_evaluate` with the non-chunked, regular `plc.io.parquet.read_parquet`: https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/dsl/ir.py#L864-L866
- Building a `filter` expression for the rapidsmpf reader: https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/streaming/actor_graph/io.py#L580-L585
- As part of `SplitScan.do_evaluate`: https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/streaming/io.py#L209-L211, to generate the `Scan` nodes dynamically
- As part of `ParquetMetadata.__init__` in stats collection (https://github.com/rapidsai/cudf/blob/a9ebd399b6102da93b8ff379f5e74ece1381e038/python/cudf_polars/cudf_polars/streaming/io.py#L515-L517)
We occasionally *re*read parquet metadata that polars has already read and parsed. https://github.com/pola-rs/polars/issues/27756 is a feature request to polars to expose some of the information it already has, which we can use when translating a `Scan` node. We can avoid rereading / reparsing the data.
One caveat: polars generally just reads a single file per table. A `scan_parquet` node might address many `paths`, but polars doesn't want to read the metadata of *every* path. But for single-file tables, we might be able to avoid rereading metadata entirely.
Contributor guide
Assessment
This issue has not been assessed yet.