NVIDIA / NVIDIA/cudf

[BUG] cuDF covariance does not agree with pandas when ddof=N

Open
#10,303 3 comments 1 reaction 0 assignees View on GitHub
bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

In PR #9889, @isVoid observed that cudf and pandas handle covariance calculations differently when `N == ddof`. Here is a minimal example showing a few differences in behavior with `ddof`. This is a bug because cudf should give the same results as pandas. The bug affects both `df.cov()` and `df.groupby(...).cov()`. Note that this is _not_ related to the pandas bug 45814 which was also found during #9889 (not directly linked here because it is not related) regarding `ddof` with missing data, because no data is missing here.

```python
import pandas as pd
import cudf

pdf = pd.DataFrame({"i": [0, 0], "a": [0, 1]})
gdf = cudf.from_pandas(pdf)
print("pandas cov")
print(pdf.cov(ddof=2))
print("cudf cov")
print(gdf.cov(ddof=2))
print("pandas groupby cov")
print(pdf.groupby("i").cov(ddof=2))
print("cudf groupby cov")
print(gdf.groupby("i").cov(ddof=2))
```

Results (warnings not shown):
```python
pandas cov
i a
i NaN NaN
a NaN inf

cudf cov
i a
i 0.0 0.0
a 0.0 0.5

pandas groupby cov
a
i
0 a inf

cudf groupby cov
a
i
0 a
```

Notice that cudf isn't self-consistent between `df.cov()` and `df.groupby(...).cov()` in its results for `a`.

_Originally posted by @bdice in https://github.com/rapidsai/cudf/pull/9889#discussion_r807267844_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.