NVIDIA / NVIDIA/cudf

Prefetch parquet metadata in the rapidsmpf streaming network

Open
#23,131 0 comments 0 reactions 1 assignee Claimed by @TomAugspurger View on GitHub
cudf-polars
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

https://github.com/rapidsai/cudf/pull/22700 adds metadata prefetching to cudf-polars. It implements it as a distinct stage that runs after prefetching and before IR execution in a rapidsmpf streaming network.

While metadata prefetching should be relatively fast compared to IR execution, the barrier created by prefetching isn't optimal. It waits for *all* metadata prefetching to *complete* before *any* of the Scan nodes can *begin*. But a Scan node really only requires the prefetched metadata for the `paths` it reads, not all the paths in the entire query.

We can fix this by removing the prefetching barrier and doing the

https://github.com/TomAugspurger/pygdf/commit/18913f5ce5155e45643124bfba3901aa90b15890 is out of date now, but has a POC for how this might be done. At a high level:

1. traverse the IR graph, grouping the Scan nodes by `paths` (we want to read each parquet metadata exactly once)
2. Create actors responsible for each group of parquet metadata
3. Feed the metadata into the Actors performing the Scans

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.