Prefetch parquet metadata in the rapidsmpf streaming network
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
https://github.com/rapidsai/cudf/pull/22700 adds metadata prefetching to cudf-polars. It implements it as a distinct stage that runs after prefetching and before IR execution in a rapidsmpf streaming network.
While metadata prefetching should be relatively fast compared to IR execution, the barrier created by prefetching isn't optimal. It waits for *all* metadata prefetching to *complete* before *any* of the Scan nodes can *begin*. But a Scan node really only requires the prefetched metadata for the `paths` it reads, not all the paths in the entire query.
We can fix this by removing the prefetching barrier and doing the
https://github.com/TomAugspurger/pygdf/commit/18913f5ce5155e45643124bfba3901aa90b15890 is out of date now, but has a POC for how this might be done. At a high level:
1. traverse the IR graph, grouping the Scan nodes by `paths` (we want to read each parquet metadata exactly once)
2. Create actors responsible for each group of parquet metadata
3. Feed the metadata into the Actors performing the Scans
Contributor guide
Assessment
This issue has not been assessed yet.