numpy / numpy/numpy

`np.isin` for int array to string array fails based on length.

Open
#18,632 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

Description

np.isin only sometimes fails when comparing integer array into string array.

Specifically, if one intended to compare to integer arrays with np.isin but second array was converted to string for some reason, it bugs out only on specific conditions relating to the length of the supplied arrays.

In my case this happened somewhere in our data pipeline missing data was converted to an empty string (''), causing the array to be converted to a string type.

I think I've figured out why it happens and already worked around the issue, but I am submitting an issue for documentation's sake.

I understand that it can be the case that np.isin working on string converted array is unintentional and should be avoided, however I believe this issue is something that can easily be encountered in the wild, and should be documented.

Reproducing code example:
import numpy as np

print(np.isin([1], [1] * 9 + ['']))  # array([True])  # works
print(np.isin([1], [1] * 8 + ['']))  # array([False])  # bugs out
Error message:

Not an error, but a warning:

{my venv folder}/lib/python3.8/site-packages/numpy/lib/arraysetops.py:583: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
mask |= (ar1 == a)

NumPy/Python version information:

1.20.0 3.8.7 (default, Mar 6 2021, 18:53:50)
[Clang 12.0.0 (clang-1200.0.32.29)]

What I think is happening

I believe this issue happens because behaviour is different from when optimization in in1d happens or not.

If optimization happens, the comparison becomes np.array([0]) == '0' which is False.

https://github.com/numpy/numpy/blob/d7aa4085623b222058edb0ff38392c38c5e00c54/numpy/lib/arraysetops.py#L575-L584

If optimization does not happen, the type is silently converted to string while doing np.concatenate((ar1, ar2)), and all values are treated as string. Therefore it returns True.

https://github.com/numpy/numpy/blob/d7aa4085623b222058edb0ff38392c38c5e00c54/numpy/lib/arraysetops.py#L591

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in numpy/lib/arraysetops.py around the referenced lines 575-584 and 591, then run the two np.isin examples from the issue to compare the optimized and non-optimized paths. Determine whether the inconsistent result should be fixed or documented; done means the behavior is consistent or its conditions and limitations are clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.