[PERF]: Cache parquet metadata at the session level
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## Problem
`prefetch_parquet_file_metadata_for_ir` fetches parquet footers from S3 at the start of every query execution, even for files that were already read in a previous iteration of the same benchmark or a previous query against the same dataset. The `CachedParquetInfo` objects (which include the parsed `FileMetaData` and pre-built `HybridScanMetadata`, and open kvikio handles as of #23317) are discarded when the query finishes.
From profiling Q1 at SF300 on a g7e.8xlarge:
It takes a second to prefetch the parquet metadata on both iterations. But the second iteration should not pay that cost.
## Proposed Fix
Add a session-level cache of `CachedParquetInfo` objects on the engine. Key the cache on `(path, etag)` where the etag is the S3 object version identifier propagated from polars via #22734.
## Related
- #22667 within-query metadata caching
- #22666 reuse metadata from polars
- #22734 object metadata (size, etag) from polars
- In PR #23317, this would give prefetching workers a larger head start to get the bytes into pinned memory. And therefore should reduce the time producers wait for byte range request futures to resolve.
Contributor guide
Assessment
This issue has not been assessed yet.