GroupBy.cov() when dask SeriesGroupBy object returns unexpected response
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the issue**:
When performing a groupby covariance on a Pandas DataFrame, where the groupby operation returns a `pd.SeriesGroupBy` object, Pandas returns a `TypeError`, which makes sense since there is not a second column. However, the same operation on a dask dataframe returns a Dask Series
**Minimal Complete Verifiable Example**:
```python
import dask.dataframe as dd
import pandas as pd
from dask.dataframe.utils import assert_eq
pdf = pd.DataFrame(
{"a": [1, 2, 6, 4, 4, 6, 4, 3, 7], "b": [4, 2, 7, 3, 3, 1, 1, 1, 2]},
index=[0, 1, 3, 5, 6, 8, 9, 9, 9],
)
ddf = dd.from_pandas(pdf, npartitions=3)
assert_eq(pdf.groupby("b").a.cov(), ddf.groupby("b").a.cov())
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [1], line 10
5 pdf = pd.DataFrame(
6 {"a": [1, 2, 6, 4, 4, 6, 4, 3, 7], "b": [4, 2, 7, 3, 3, 1, 1, 1, 2]},
7 index=[0, 1, 3, 5, 6, 8, 9, 9, 9],
8 )
9 ddf = dd.from_pandas(pdf, npartitions=3)
---> 10 assert_eq(pdf.groupby("b").a.cov(), ddf.groupby("b").a.cov())
File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:950, in GroupBy._make_wrapper..wrapper(*args, **kwargs)
947 if name in base.plotting_methods:
948 return self.apply(curried)
--> 950 return self._python_apply_general(curried, self._obj_with_exclusions)
File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:1464, in GroupBy._python_apply_general(self, f, data, not_indexed_same)
1438 @final
1439 def _python_apply_general(
1440 self,
(...)
1443 not_indexed_same: bool | None = None,
1444 ) -> DataFrame | Series:
1445 """
1446 Apply function f in python space
1447
(...)
1462 data after applying f
1463 """
-> 1464 values, mutated = self.grouper.apply(f, data, self.axis)
1466 if not_indexed_same is None:
1467 not_indexed_same = mutated or self.mutated
File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/ops.py:761, in BaseGrouper.apply(self, f, data, axis)
759 # group might be modified
760 group_axes = group.axes
--> 761 res = f(group)
762 if not mutated and not _is_indexed_like(res, group_axes, axis):
763 mutated = True
File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:939, in GroupBy._make_wrapper..wrapper..curried(x)
938 def curried(x):
--> 939 return f(x, *args, **kwargs)
TypeError: cov() missing 1 required positional argument: 'other'
```
See:
```
ddf.groupby("b").a.cov().compute()
b
1 a 2.333333
2 a 12.500000
3 a 0.000000
4 a NaN
7 a NaN
Name: a, dtype: float64
```
```
pdf.groupby("b").a.cov().compute()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [4], line 1
----> 1 pdf.groupby("b").a.cov()
File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:950, in GroupBy._make_wrapper..wrapper(*args, **kwargs)
947 if name in base.plotting_methods:
948 return self.apply(curried)
--> 950 return self._python_apply_general(curried, self._obj_with_exclusions)
File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:1464, in GroupBy._python_apply_general(self, f, data, not_indexed_same)
1438 @final
1439 def _python_apply_general(
1440 self,
(...)
1443 not_indexed_same: bool | None = None,
1444 ) -> DataFrame | Series:
1445 """
1446 Apply function f in python space
1447
(...)
1462 data after applying f
1463 """
-> 1464 values, mutated = self.grouper.apply(f, data, self.axis)
1466 if not_indexed_same is None:
1467 not_indexed_same = mutated or self.mutated
File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/ops.py:761, in BaseGrouper.apply(self, f, data, axis)
759 # group might be modified
760 group_axes = group.axes
--> 761 res = f(group)
762 if not mutated and not _is_indexed_like(res, group_axes, axis):
763 mutated = True
File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:939, in GroupBy._make_wrapper..wrapper..curried(x)
938 def curried(x):
--> 939 return f(x, *args, **kwargs)
TypeError: cov() missing 1 required positional argument: 'other'
```
**Environment**:
- Dask version: 2022.10.1
- Python version: 3.9
- Operating System: MacOs
Contributor guide
Assessment
This issue has not been assessed yet.