[Story]: Adopt hybrid scan reader in cudf-polars to enable asynchronous prefetching of byte ranges to pinned host memory
Open
cudf-polars
cuIO
feature request
Performance
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Hybrid scan lets us split parquet IO into separate stages so we can overlap IO with GPU compute.
This is especially valuable for remote storage (S3) where IO latency is high and being able to get data into pinned host memory ahead of time significantly reduces the time producers spend waiting.
Contributor guide
Assessment
This issue has not been assessed yet.