[FEA] Support Polars pearson correlation expression
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
We should support Pearson correlation in the Polars executor for both columns and grouped objects.
```python
import polars as pl
from functools import partial
from cudf_polars.callback import execute_with_cudf
use_cudf = partial(execute_with_cudf, raise_on_fail=True) # for testing
df = pl.LazyFrame({'a': [1, 1, 2, 2], 'b': [1, 2, 3, 4], 'c': [1, 2, -3, -4]})
print(df.select(pl.corr(pl.col('b'), pl.col('c'))).collect())
print(df.select(pl.corr(pl.col('b'), pl.col('c'))).collect(post_opt_callback=use_cudf))
shape: (1, 1)
┌───────────┐
│ b │
│ --- │
│ f64 │
╞═══════════╡
│ -0.877058 │
└───────────┘
---------------------------------------------------------------------------
ComputeError Traceback (most recent call last)
Cell In[21], line 10
7 df = pl.LazyFrame({'a': [1, 1, 2, 2], 'b': [1, 2, 3, 4], 'c': [1, 2, -3, -4]})
9 print(df.select(pl.corr(pl.col('b'), pl.col('c'))).collect())
---> 10 print(df.select(pl.corr(pl.col('b'), pl.col('c'))).collect(post_opt_callback=use_cudf))
File [/raid/nicholasb/miniconda3/envs/all_cuda-122_arch-x86_64/lib/python3.11/site-packages/polars/lazyframe/frame.py:1942](http://10.117.23.184:8882/lab/tree/raid/nicholasb/raid/nicholasb/miniconda3/envs/all_cuda-122_arch-x86_64/lib/python3.11/site-packages/polars/lazyframe/frame.py#line=1941), in LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, no_optimization, streaming, background, _eager, **_kwargs)
1939 # Only for testing purposes atm.
1940 callback = _kwargs.get("post_opt_callback")
-> 1942 return wrap_df(ldf.collect(callback))
ComputeError: 'cuda' conversion failed: NotImplementedError: corr
```
Contributor guide
Assessment
This issue has not been assessed yet.