NVIDIA / NVIDIA/cudf

[FEA] Propagate nulls through `isin`

Open
#7,556 3 comments 0 reactions 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
In pandas, we can check if the values of a series or dataframe are contained within some other container, like a list or dataframe, by using `isin`. Currently, this doesn't work correctly for nulls. On branch-0.19, if the dataframe or series we're checking contains an ``, we get a `False`:

```
>>> values = cudf.Series([1,2,3])
>>> df = cudf.DataFrame({'a':[1,2,None]})
>>> df
a
0 1
1 2
2
>>> df.isin(values)
a
0 True
1 True
2 False
```

Where we should get just another `` there, like in pandas, using nullable dtypes:

```
>>> values = pd.Series([1,2,3], dtype='Int64')
>>> df = pd.DataFrame({'a':pd.Series([1,2,None], dtype='Int64')})
>>> df
a
0 1
1 2
2
>>> df.isin(values)
a
0 True
1 True
2
```

While the `fillna` that causes us to get `False` is being removed in PR https://github.com/rapidsai/cudf/pull/7490, we'll need to rework how we're testing this functionality and change it to test against nullable types. It just so happens that when using non nullable pandas types, we get `False` as well - hence our results lining up so far.

**Describe the solution you'd like**
We should get an `` everywhere the series or dataframe in question already has an `` and our tests should be updated to reflect that.

**Describe alternatives you've considered**
We could change it as part of PR https://github.com/rapidsai/cudf/pull/7490 but it would be somewhat tangential to the point.

**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.