[FEA] Support Correlation with Nullable DataFrames
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Today computing correlation for a dataframe with missing values is not possible, but pandas can do it:
```python
In [10]: gdf = cudf.DataFrame({"a": [1, None, 1, 1, None, 2, 2], "b": range(7)})
In [11]: gdf.corr()
...
ValueError: Column must have no nulls.
In [14]: pdf = pd.DataFrame({"a": [1, None, 1, 1, None, 2, 2], "b": range(7)})
In [15]: pdf.corr()
Out[15]:
a b
a 1.000000 0.879427
b 0.879427 1.000000
```
**Describe the solution you'd like**
We should examine the techniques that pandas used here. In groupby cov/corr's case, we found that pandas uses pairwise deletion on missing values that may result in a non-PSD matrix. There's a test case to make sure cuDF matches with pandas in these situations. Here we should do the same.
**Describe alternatives you've considered**
There has been [discussions](https://github.com/rapidsai/cudf/pull/9889#discussion_r808439168) about merging groupby correlation/covariance with regular corr/cov so that they share the same algorithm backbone. But I don't think we need to achieve this in one big step. Extending existing function to support more inputs is a good start.
Contributor guide
Assessment
This issue has not been assessed yet.