NVIDIA / NVIDIA/cudf

[Story]: Adopt hybrid scan reader in cudf-polars to enable asynchronous prefetching of byte ranges to pinned host memory

Open
#22,752 1 comment 0 reactions 2 assignees Claimed by @TomAugspurger View on GitHub
cudf-polars cuIO feature request Performance
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Hybrid scan lets us split parquet IO into separate stages so we can overlap IO with GPU compute.

This is especially valuable for remote storage (S3) where IO latency is high and being able to get data into pinned host memory ahead of time significantly reduces the time producers spend waiting.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.