dask / dask/dask

GroupBy.cov() when dask SeriesGroupBy object returns unexpected response

Open
#9,611 1 comment 0 reactions 0 assignees View on GitHub
bug dataframe needs attention
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:

When performing a groupby covariance on a Pandas DataFrame, where the groupby operation returns a `pd.SeriesGroupBy` object, Pandas returns a `TypeError`, which makes sense since there is not a second column. However, the same operation on a dask dataframe returns a Dask Series

**Minimal Complete Verifiable Example**:

```python

import dask.dataframe as dd
import pandas as pd
from dask.dataframe.utils import assert_eq

pdf = pd.DataFrame(
{"a": [1, 2, 6, 4, 4, 6, 4, 3, 7], "b": [4, 2, 7, 3, 3, 1, 1, 1, 2]},
index=[0, 1, 3, 5, 6, 8, 9, 9, 9],
)
ddf = dd.from_pandas(pdf, npartitions=3)
assert_eq(pdf.groupby("b").a.cov(), ddf.groupby("b").a.cov())
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [1], line 10
5 pdf = pd.DataFrame(
6 {"a": [1, 2, 6, 4, 4, 6, 4, 3, 7], "b": [4, 2, 7, 3, 3, 1, 1, 1, 2]},
7 index=[0, 1, 3, 5, 6, 8, 9, 9, 9],
8 )
9 ddf = dd.from_pandas(pdf, npartitions=3)
---> 10 assert_eq(pdf.groupby("b").a.cov(), ddf.groupby("b").a.cov())

File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:950, in GroupBy._make_wrapper..wrapper(*args, **kwargs)
947 if name in base.plotting_methods:
948 return self.apply(curried)
--> 950 return self._python_apply_general(curried, self._obj_with_exclusions)

File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:1464, in GroupBy._python_apply_general(self, f, data, not_indexed_same)
1438 @final
1439 def _python_apply_general(
1440 self,
(...)
1443 not_indexed_same: bool | None = None,
1444 ) -> DataFrame | Series:
1445 """
1446 Apply function f in python space
1447
(...)
1462 data after applying f
1463 """
-> 1464 values, mutated = self.grouper.apply(f, data, self.axis)
1466 if not_indexed_same is None:
1467 not_indexed_same = mutated or self.mutated

File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/ops.py:761, in BaseGrouper.apply(self, f, data, axis)
759 # group might be modified
760 group_axes = group.axes
--> 761 res = f(group)
762 if not mutated and not _is_indexed_like(res, group_axes, axis):
763 mutated = True

File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:939, in GroupBy._make_wrapper..wrapper..curried(x)
938 def curried(x):
--> 939 return f(x, *args, **kwargs)

TypeError: cov() missing 1 required positional argument: 'other'
```
See:
```
ddf.groupby("b").a.cov().compute()
b
1 a 2.333333
2 a 12.500000
3 a 0.000000
4 a NaN
7 a NaN
Name: a, dtype: float64
```
```
pdf.groupby("b").a.cov().compute()

---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [4], line 1
----> 1 pdf.groupby("b").a.cov()

File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:950, in GroupBy._make_wrapper..wrapper(*args, **kwargs)
947 if name in base.plotting_methods:
948 return self.apply(curried)
--> 950 return self._python_apply_general(curried, self._obj_with_exclusions)

File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:1464, in GroupBy._python_apply_general(self, f, data, not_indexed_same)
1438 @final
1439 def _python_apply_general(
1440 self,
(...)
1443 not_indexed_same: bool | None = None,
1444 ) -> DataFrame | Series:
1445 """
1446 Apply function f in python space
1447
(...)
1462 data after applying f
1463 """
-> 1464 values, mutated = self.grouper.apply(f, data, self.axis)
1466 if not_indexed_same is None:
1467 not_indexed_same = mutated or self.mutated

File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/ops.py:761, in BaseGrouper.apply(self, f, data, axis)
759 # group might be modified
760 group_axes = group.axes
--> 761 res = f(group)
762 if not mutated and not _is_indexed_like(res, group_axes, axis):
763 mutated = True

File ~/mambaforge/envs/dask/lib/python3.9/site-packages/pandas/core/groupby/groupby.py:939, in GroupBy._make_wrapper..wrapper..curried(x)
938 def curried(x):
--> 939 return f(x, *args, **kwargs)

TypeError: cov() missing 1 required positional argument: 'other'
```

**Environment**:

- Dask version: 2022.10.1
- Python version: 3.9
- Operating System: MacOs

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.