NVIDIA / NVIDIA/cudf

[Story] Improve cudf-polars remote IO

Open
#22,662 4 comments 3 reactions 2 assignees Claimed by @Matt711 View on GitHub
cudf-polars feature request Performance
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

This issue summarize some performance issues we're running into with (remote) IO in cudf-polars, and some potential fixes. In this issue, we're primarily focused on remote IO (high latency / high throughput storage systems) from parquet sources, but some of these lessons apply to local IO and non-parquet sources.

## Work Streams

At a high-level, we can break the work into 3 (mostly) independent work streams: optimizing libcudf (and kvikio), getting more information out of polars and using it in cudf-polars, and optimizing cudf-polars.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.