[FEA] Propagate nulls through `isin`
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
In pandas, we can check if the values of a series or dataframe are contained within some other container, like a list or dataframe, by using `isin`. Currently, this doesn't work correctly for nulls. On branch-0.19, if the dataframe or series we're checking contains an ``, we get a `False`:
```
>>> values = cudf.Series([1,2,3])
>>> df = cudf.DataFrame({'a':[1,2,None]})
>>> df
a
0 1
1 2
2
>>> df.isin(values)
a
0 True
1 True
2 False
```
Where we should get just another `` there, like in pandas, using nullable dtypes:
```
>>> values = pd.Series([1,2,3], dtype='Int64')
>>> df = pd.DataFrame({'a':pd.Series([1,2,None], dtype='Int64')})
>>> df
a
0 1
1 2
2
>>> df.isin(values)
a
0 True
1 True
2
```
While the `fillna` that causes us to get `False` is being removed in PR https://github.com/rapidsai/cudf/pull/7490, we'll need to rework how we're testing this functionality and change it to test against nullable types. It just so happens that when using non nullable pandas types, we get `False` as well - hence our results lining up so far.
**Describe the solution you'd like**
We should get an `` everywhere the series or dataframe in question already has an `` and our tests should be updated to reflect that.
**Describe alternatives you've considered**
We could change it as part of PR https://github.com/rapidsai/cudf/pull/7490 but it would be somewhat tangential to the point.
**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.