GroupBy.idxmin and idxmax are very broken
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
The implementation is broken as soon as the max or min value is not in the first partition
This was broken for years, so only adding for API compatibility now but we should fix this
```
def test_df_groupby_idxmax():
pdf = pd.DataFrame(
{"idx": list(range(4)), "group": [1, 1, 2, 2, 1], "value": [10, 20, 20, 10, 40]}
).set_index("idx")
ddf = dd.from_pandas(pdf, npartitions=3)
expected = pd.DataFrame({"group": [1, 2], "value": [4, 2]}).set_index("group")
result_pd = pdf.groupby("group").idxmax()
result_dd = ddf.groupby("group").idxmax()
assert_eq(result_pd, result_dd)
assert_eq(expected, result_dd)
```
This is a very simple reproducer for the underlying issue
Contributor guide
Research direction
Start by running the provided test_df_groupby_idxmax reproducer with three partitions and compare the Dask result with pandas and the expected DataFrame. Trace the GroupBy.idxmax and idxmin entry points to find how partition results are combined; done means extrema are found even when the max or min is not in the first partition, with results matching pandas.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100