NVIDIA / NVIDIA/cudf

[FEA] Support Polars pearson correlation expression

Open
#16,220 3 comments 0 reactions 0 assignees View on GitHub
cudf-polars feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

We should support Pearson correlation in the Polars executor for both columns and grouped objects.

```python
import polars as pl
from functools import partial
from cudf_polars.callback import execute_with_cudf

use_cudf = partial(execute_with_cudf, raise_on_fail=True) # for testing

df = pl.LazyFrame({'a': [1, 1, 2, 2], 'b': [1, 2, 3, 4], 'c': [1, 2, -3, -4]})

print(df.select(pl.corr(pl.col('b'), pl.col('c'))).collect())
print(df.select(pl.corr(pl.col('b'), pl.col('c'))).collect(post_opt_callback=use_cudf))
shape: (1, 1)
┌───────────┐
│ b │
│ --- │
│ f64 │
╞═══════════╡
│ -0.877058 │
└───────────┘
---------------------------------------------------------------------------
ComputeError Traceback (most recent call last)
Cell In[21], line 10
7 df = pl.LazyFrame({'a': [1, 1, 2, 2], 'b': [1, 2, 3, 4], 'c': [1, 2, -3, -4]})
9 print(df.select(pl.corr(pl.col('b'), pl.col('c'))).collect())
---> 10 print(df.select(pl.corr(pl.col('b'), pl.col('c'))).collect(post_opt_callback=use_cudf))

File [/raid/nicholasb/miniconda3/envs/all_cuda-122_arch-x86_64/lib/python3.11/site-packages/polars/lazyframe/frame.py:1942](http://10.117.23.184:8882/lab/tree/raid/nicholasb/raid/nicholasb/miniconda3/envs/all_cuda-122_arch-x86_64/lib/python3.11/site-packages/polars/lazyframe/frame.py#line=1941), in LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, no_optimization, streaming, background, _eager, **_kwargs)
1939 # Only for testing purposes atm.
1940 callback = _kwargs.get("post_opt_callback")
-> 1942 return wrap_df(ldf.collect(callback))

ComputeError: 'cuda' conversion failed: NotImplementedError: corr
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.