NVIDIA / NVIDIA/cudf

[BUG] Treatment of logical and bitwise binops in `DataFrame.eval` does not match pandas

Open
#14,517 3 comments 0 reactions 0 assignees View on GitHub
bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

In pandas, `eval` treats `a op b` as always meaning the bitwise version `op in {and, or, &, |, ^}`. In cudf, due to the way we parse the expression without type information (and the dispatching scheme to the AST interpreter in libcudf), `and` and `or` mean "logical" and `&`, `|`, and `^` mean "bitwise". There's a final wrinkle that (like spark) for bools only, masked values are treated as `False`.

This can cause differences in the result between calling `eval` with pandas and with cudf. Although the docstring mentions these differences, when using `cudf.pandas`, we don't see the cudf docstring (only the pandas one).

**Expected behavior**

Eventually, we should match pandas. I think this should be done by running a type inference pass on the user-provided expression and rewriting to an appropriate combination of bitwise and logical operations. This would have the nice side-effect of also allowing mixed-type operands in `eval` expressions by cudf upcasting before passing off to libcudf.

In the short term, we should probably raise a `NotImplementedError` when running in pandas-compat mode if the expression contains logical/bitwise binops.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.