NVIDIA / NVIDIA/cudf

[Story] Sink support in cudf-polars

Open
#18,567 0 comments 0 reactions 1 assignee Claimed by @Matt711 View on GitHub
cudf-polars feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

# Background

`Sink` support in the GPU engine has been a journey, and it certainly has not had the same time dedicated to as `Scan`. This primarily due to `Scan` being more important to our benchmarking story in cudf-polars. Which has primarily consisted of the TPC-like benchmarks PDS-H and PDS-DS. But `Sink` is also very important data processing ecosystem becomes it enables sharing results with others or storing results for later. So we can't for get the "S" in ETLS; it's for Store/Share/Sink!

# Current State
PRs
- [x] Sink support in the in-memory GPU engine https://github.com/rapidsai/cudf/pull/18468
- [x] Use libcudf chunked parquert writer in cudf-polars: https://github.com/rapidsai/cudf/issues/18969
- [x] single-file streaming Sink support https://github.com/rapidsai/cudf/pull/18963

The main thing left TODO is to support `pl.PartitionBy` or hive-style partitioned sinks in both streaming and in-memory GPU engines.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.