NVIDIA / NVIDIA/cudf

Experiment with cccl.parallel and cudf-polars

Open
#19,947 0 comments 1 reaction 0 assignees View on GitHub
cudf-polars proposal Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

In principle it should be possible to use cccl.parallel with pylibcudf today. Where it is likely to have the greatest impact is in operator fusion, i.e. bypassing multiple libcudf calls in favor of using a single cccl.parallel call that performs multiple operations. In the long run, I think cccl.parallel is probably the wrong layer at which for us to interact with cccl since we probably want any sort of fusion that we support to also be supported in the other front-ends for libcudf, including in other languages (i.e. Spark RAPIDS, Velox, etc). Improving JIT compilation a la https://github.com/rapidsai/cudf/issues/15366 is how we would handle that. However, there is a long way to go before our JIT functionality in libcudf is mature enough to support a lot of what we want, so in the meantime cccl.parallel provides a very nice testing ground for us to determine when operator fusion is possible/useful and can also help the cccl Python team get some immediate feedback on the library as they develop.

The pandas API probably won't be the best testing ground since it is an eager API and we would have to jump through tons of hoops to try and make it appear lazy (something we've already always avoided). pylibcudf is too low-level since it is already effectively an operator library that implements one operation at a time. Therefore, I think cudf-polars is probably the right place to start these experiments. With cudf-polars, it should be straightforward to build out some experimental code that replace pylibcudf-based execution nodes with cccl.parallel-based execution nodes. Making it work in general for any operator will probably be out of scope since it requires a full subtree traversal to ensure that the entire subexpression can be handled by cccl.parallel, but for now we can make some assumptions about e.g. subtree depth to make things easier. In principle we could even put the results into production if we see good outcomes by doing a quick pass of subtree traversal for a conditional determination of whether to use cccl.parallel or not.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.