NVIDIA / NVIDIA/cudf

[PERF]: Cache parquet metadata at the session level

Open
#23,526 1 comment 0 reactions 1 assignee Claimed by @Matt711 View on GitHub
cudf-polars Performance
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Problem

`prefetch_parquet_file_metadata_for_ir` fetches parquet footers from S3 at the start of every query execution, even for files that were already read in a previous iteration of the same benchmark or a previous query against the same dataset. The `CachedParquetInfo` objects (which include the parsed `FileMetaData` and pre-built `HybridScanMetadata`, and open kvikio handles as of #23317) are discarded when the query finishes.

From profiling Q1 at SF300 on a g7e.8xlarge:

Image

It takes a second to prefetch the parquet metadata on both iterations. But the second iteration should not pay that cost.

## Proposed Fix

Add a session-level cache of `CachedParquetInfo` objects on the engine. Key the cache on `(path, etag)` where the etag is the S3 object version identifier propagated from polars via #22734.

## Related

- #22667 within-query metadata caching
- #22666 reuse metadata from polars
- #22734 object metadata (size, etag) from polars
- In PR #23317, this would give prefetching workers a larger head start to get the bytes into pinned memory. And therefore should reduce the time producers wait for byte range request futures to resolve.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.