dask / dask/dask-expr

GroupBy.idxmin and idxmax are very broken

Open
#584 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

The implementation is broken as soon as the max or min value is not in the first partition

This was broken for years, so only adding for API compatibility now but we should fix this

```
def test_df_groupby_idxmax():
pdf = pd.DataFrame(
{"idx": list(range(4)), "group": [1, 1, 2, 2, 1], "value": [10, 20, 20, 10, 40]}
).set_index("idx")

ddf = dd.from_pandas(pdf, npartitions=3)

expected = pd.DataFrame({"group": [1, 2], "value": [4, 2]}).set_index("group")

result_pd = pdf.groupby("group").idxmax()
result_dd = ddf.groupby("group").idxmax()

assert_eq(result_pd, result_dd)
assert_eq(expected, result_dd)
```

This is a very simple reproducer for the underlying issue

Contributor guide

Open the contributing guide

Research direction

Start by running the provided test_df_groupby_idxmax reproducer with three partitions and compare the Dask result with pandas and the expected DataFrame. Trace the GroupBy.idxmax and idxmin entry points to find how partition results are combined; done means extrema are found even when the max or min is not in the first partition, with results matching pandas.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.