[FEA] Support floating dtypes for `corr` aggregations in JIT GroupBy Apply
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Currently only [integer dtypes](https://github.com/rapidsai/cudf/blob/branch-23.10/python/cudf/cudf/core/udf/groupby_typing.py#L257-L258) are supported when applying a correlation aggregation between columns in JIT GroupBy apply. This is mainly lacking due to requiring special treatment to match pandas treatment for special values. A note in the [pandas docs](https://pandas.pydata.org/docs/reference/api/pandas.Series.corr.html) says:
```
Compute correlation with other Series, excluding missing values.
```
In general, JIT groupby apply does not support nulls, but it seems that nans are covered under this umbrella as well. By "excluding" missing values, pandas means that the final correlation coefficient is the same as one which would have been computed if all pairs of datapoints where either value is nan is excluded from the inner sum of the algorithm. This can be seen as follows:
```python
>>> sr1 = pd.Series([1.0, 2.0, float('nan'), 4.0, float('nan'), 6.0])
>>> sr2 = pd.Series([1.1, 1.9, 3.0, float('nan'), float('nan'), 6.7])
>>> mask = ~(sr1.isna() | sr2.isna())
>>> sr1.corr(sr2)
0.9983374884595826
>>> sr1[mask].corr(sr2[mask])
0.9983374884595826
```
**Describe the solution you'd like**
I'd like this to work and return the same value as pandas. This requires some additional engineering inside of our shim library to support this kind of element deletion on the c++ side. Currently `corr` leverages various other block level functions such as covariance to compute the pearson correlation. We'd have to figure out if we can adapt those functions to support this case or if we'd have to do something more involved for `corr` specifically.
Contributor guide
Assessment
This issue has not been assessed yet.