[BUG] Treatment of logical and bitwise binops in `DataFrame.eval` does not match pandas
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
In pandas, `eval` treats `a op b` as always meaning the bitwise version `op in {and, or, &, |, ^}`. In cudf, due to the way we parse the expression without type information (and the dispatching scheme to the AST interpreter in libcudf), `and` and `or` mean "logical" and `&`, `|`, and `^` mean "bitwise". There's a final wrinkle that (like spark) for bools only, masked values are treated as `False`.
This can cause differences in the result between calling `eval` with pandas and with cudf. Although the docstring mentions these differences, when using `cudf.pandas`, we don't see the cudf docstring (only the pandas one).
**Expected behavior**
Eventually, we should match pandas. I think this should be done by running a type inference pass on the user-provided expression and rewriting to an appropriate combination of bitwise and logical operations. This would have the nice side-effect of also allowing mixed-type operands in `eval` expressions by cudf upcasting before passing off to libcudf.
In the short term, we should probably raise a `NotImplementedError` when running in pandas-compat mode if the expression contains logical/bitwise binops.
Contributor guide
Assessment
This issue has not been assessed yet.