NVIDIA / NVIDIA/cudf

[BUG] Nondeterministic behavior in groupby-idxmax/idxmin operations

Open
#17,479 9 comments 0 reactions 0 assignees View on GitHub
bug libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
I'm not sure if this is a true bug, but I _sometimes_ get different results after doing a groupby-idxmax operation. I'd appreciate it if someone else can reproduce too.

**Steps/Code to reproduce bug**
I haven't narrowed the bug down any further, but I've been able to reproduce the bug when `n=200` in the snippet below.
```python
import cudf
from cudf.testing import assert_frame_equal
df = cudf.DataFrame({
"a": [4, 0, 2, 2, 1, 4, 4, 4, 2, 1, 1, 1, 3, 3, 4, 1, 4, 4, 1, 0, 1, 2, 3, 4, 4, 4, 1, 2, 3, 2, 4, 2, 2, 2, 2, 2, 2, 3, 2, 1, 2, 2, 0, 2, 0, 3, 3, 0, 1, 2],
"b": [1, 1, 4, 0, 3, 3, 0, 0, 1, 3, 2, 0, 2, 3, 2, 3, 4, 3, 1, 0, 1, 2, 4, 3, 1, 4, 2, 3, 0, 0, 1, 4, 3, 4, 3, 2, 2, 2, 4, 2, 4, 3, 4, 2, 3, 1, 2, 1, 0, 0]
})

n=200
for _ in range(n):
df1 = df.groupby("a").idxmax()
df2 = df.groupby("a").idxmax()
assert_frame_equal(df1, df2) # true for most iterations (but sometimes fails)
```
```
AssertionError: Column name="b" are different

values are different (100.0 %)
[left]: {'[np.int64(42), np.int64(33), np.int64(16), np.int64(4), np.int64(22)]'}
[right]: {'[np.int64(42), np.int64(2), np.int64(16), np.int64(4), np.int64(22)]'}
```

**Expected behavior**
I expect the dataframes to be the same.

**Additional context**
I found this by collecting fallback logs in the groupby tests in the pandas test suite.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.