NVIDIA / NVIDIA/cudf

[BUG] Incorrect `NaN` ignoring in GroupBy JIT Reductions

Open
#14,860 1 comment 0 reactions 1 assignee Claimed by @brandon-b-miller View on GitHub
bug numba Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Pandas series and dataframe reductions default to `skipna=True`, our JIT operators don't fully support this behavior.

**Steps/Code to reproduce bug**
```python
import cudf
df = cudf.DataFrame({
'key1': [0,0],
'val1':[1, float('nan')]
}, nan_as_null=False)

def func(df):
return df['val1'].sum()

expect = df.to_pandas().groupby('key1').apply(func)
got = df.groupby('key1').apply(func, engine='jit')

print(expect)
print(got)
```
```
key1
0 1.0
dtype: float64
key1
0 NaN
dtype: float64
```

**Expected behavior**
`NaN` values should be ignored by default and the same numerical value should be returned from cuDF as pandas.

**Environment overview (please complete the following information)**
- Environment location: Bare-metal
- Method of cuDF install: Source

**Additional context**
The current operators have _some_ `NaN` handling, but it's mostly to cover cases where the entire sequence of values is `NaN` or for edge cases in some operators where we'd end up dividing by zero. However general `NaN` _skipping_ is not implemented leading to the buggy behavior.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.