[Story] Improve cudf-polars remote IO
Open
cudf-polars
feature request
Performance
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
This issue summarize some performance issues we're running into with (remote) IO in cudf-polars, and some potential fixes. In this issue, we're primarily focused on remote IO (high latency / high throughput storage systems) from parquet sources, but some of these lessons apply to local IO and non-parquet sources.
## Work Streams
At a high-level, we can break the work into 3 (mostly) independent work streams: optimizing libcudf (and kvikio), getting more information out of polars and using it in cudf-polars, and optimizing cudf-polars.
Contributor guide
Assessment
This issue has not been assessed yet.