NVIDIA / NVIDIA/cudf

[BUG] Inconsistent Results Between AST and Pandas Evaluators for Expressions with Nullable Columns

Open
#14,361 4 comments 0 reactions 0 assignees View on GitHub
0 - Backlog bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Describe the bug

The AST evaluator and pandas evaluator are yielding divergent outcomes when evaluating expressions that include nullable columns.

Steps/Code to reproduce bug

```
import cudf
import pandas as pd
from io import StringIO

# Your CSV data as a string
csv_data = """\
Brand#32,MED PKG,,,754.84,4428,3537.78
Brand#53,MED BOX,,,3.98,1646,23333.44
Brand#41,WRAP BAG,1,8,763.34,3547,14687.36
Brand#43,SM BOX,2,46,15.45,6523,72572.36
Brand#42,WRAP PKG,3,31,778.19,3849,33608.96
Brand#23,LG JAR,4,43,101.93,4592,50407.61
"""
dtype_schema = {
'p_brand': 'string',
'p_container': 'string',
'l_linenumber': 'float64',
'l_quantity': 'float64',
'ps_supplycost': 'float64',
'ps_availqty': 'float64',
'l_extendedprice': 'float64'
}

df = pd.read_csv(StringIO(csv_data), header=None, delimiter=',', dtype=dtype_schema, names=[name for name in dtype_schema])

df.eval("(p_brand == 'Brand#23' or p_container == 'MED BOX') or (l_quantity < ps_supplycost)").value_counts()
gdf = cudf.from_pandas(df)

gdf.eval("(p_brand == 'Brand#23' or p_container == 'MED BOX') or (l_quantity < ps_supplycost)").value_counts()
```

## Expected behavior

The cuDF AST evaluator is expected to yield results consistent with those produced by the pandas evaluator.

See:
```
>>> df.eval("(p_brand == 'Brand#23' or p_container == 'MED BOX') or (l_quantity < ps_supplycost)").value_counts()
True 4
False 2
dtype: int64

>>> gdf = cudf.from_pandas(df)
>>> gdf.eval("(p_brand == 'Brand#23' or p_container == 'MED BOX') or (l_quantity < ps_supplycost)").value_counts()
True 3
False 1
dtype: int32

>>> df[ ((df['p_brand'] == 'Brand#23') | (df['p_container'] == 'MED BOX')) | (df['l_quantity'] < df['ps_supplycost']) ]
p_brand p_container l_linenumber l_quantity ps_supplycost ps_availqty l_extendedprice
1 Brand#53 MED BOX NaN NaN 3.98 1646.0 23333.44
2 Brand#41 WRAP BAG 1.0 8.0 763.34 3547.0 14687.36
4 Brand#42 WRAP PKG 3.0 31.0 778.19 3849.0 33608.96
5 Brand#23 LG JAR 4.0 43.0 101.93 4592.0 50407.61
>>> gdf[ ((gdf['p_brand'] == 'Brand#23') | (gdf['p_container'] == 'MED BOX')) | (gdf['l_quantity'] < gdf['ps_supplycost']) ]
p_brand p_container l_linenumber l_quantity ps_supplycost ps_availqty l_extendedprice
2 Brand#41 WRAP BAG 1.0 8.0 763.34 3547.0 14687.36
4 Brand#42 WRAP PKG 3.0 31.0 778.19 3849.0 33608.96
5 Brand#23 LG JAR 4.0 43.0 101.93 4592.0 50407.61
>>>
```

Environment overview

Environment location: conda
Method of cuDF install: conda
branch-23.10, origin/branch-23.10

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.