NVIDIA / NVIDIA/cudf

[FEA] Support Correlation with Nullable DataFrames

Open
#10,424 9 comments 0 reactions 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
Today computing correlation for a dataframe with missing values is not possible, but pandas can do it:
```python
In [10]: gdf = cudf.DataFrame({"a": [1, None, 1, 1, None, 2, 2], "b": range(7)})
In [11]: gdf.corr()
...
ValueError: Column must have no nulls.

In [14]: pdf = pd.DataFrame({"a": [1, None, 1, 1, None, 2, 2], "b": range(7)})
In [15]: pdf.corr()
Out[15]:
a b
a 1.000000 0.879427
b 0.879427 1.000000
```

**Describe the solution you'd like**
We should examine the techniques that pandas used here. In groupby cov/corr's case, we found that pandas uses pairwise deletion on missing values that may result in a non-PSD matrix. There's a test case to make sure cuDF matches with pandas in these situations. Here we should do the same.

**Describe alternatives you've considered**
There has been [discussions](https://github.com/rapidsai/cudf/pull/9889#discussion_r808439168) about merging groupby correlation/covariance with regular corr/cov so that they share the same algorithm backbone. But I don't think we need to achieve this in one big step. Extending existing function to support more inputs is a good start.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.