[BUG] cuDF covariance does not agree with pandas when ddof=N
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
In PR #9889, @isVoid observed that cudf and pandas handle covariance calculations differently when `N == ddof`. Here is a minimal example showing a few differences in behavior with `ddof`. This is a bug because cudf should give the same results as pandas. The bug affects both `df.cov()` and `df.groupby(...).cov()`. Note that this is _not_ related to the pandas bug 45814 which was also found during #9889 (not directly linked here because it is not related) regarding `ddof` with missing data, because no data is missing here.
```python
import pandas as pd
import cudf
pdf = pd.DataFrame({"i": [0, 0], "a": [0, 1]})
gdf = cudf.from_pandas(pdf)
print("pandas cov")
print(pdf.cov(ddof=2))
print("cudf cov")
print(gdf.cov(ddof=2))
print("pandas groupby cov")
print(pdf.groupby("i").cov(ddof=2))
print("cudf groupby cov")
print(gdf.groupby("i").cov(ddof=2))
```
Results (warnings not shown):
```python
pandas cov
i a
i NaN NaN
a NaN inf
cudf cov
i a
i 0.0 0.0
a 0.0 0.5
pandas groupby cov
a
i
0 a inf
cudf groupby cov
a
i
0 a
```
Notice that cudf isn't self-consistent between `df.cov()` and `df.groupby(...).cov()` in its results for `a`.
_Originally posted by @bdice in https://github.com/rapidsai/cudf/pull/9889#discussion_r807267844_
Contributor guide
Assessment
This issue has not been assessed yet.