`np.isin` for int array to string array fails based on length.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
Description
np.isin only sometimes fails when comparing integer array into string array.
Specifically, if one intended to compare to integer arrays with np.isin but second array was converted to string for some reason, it bugs out only on specific conditions relating to the length of the supplied arrays.
In my case this happened somewhere in our data pipeline missing data was converted to an empty string (''), causing the array to be converted to a string type.
I think I've figured out why it happens and already worked around the issue, but I am submitting an issue for documentation's sake.
I understand that it can be the case that np.isin working on string converted array is unintentional and should be avoided, however I believe this issue is something that can easily be encountered in the wild, and should be documented.
Reproducing code example:
import numpy as np
print(np.isin([1], [1] * 9 + [''])) # array([True]) # works
print(np.isin([1], [1] * 8 + [''])) # array([False]) # bugs out
Error message:
Not an error, but a warning:
{my venv folder}/lib/python3.8/site-packages/numpy/lib/arraysetops.py:583: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
mask |= (ar1 == a)
NumPy/Python version information:
1.20.0 3.8.7 (default, Mar 6 2021, 18:53:50)
[Clang 12.0.0 (clang-1200.0.32.29)]
What I think is happening
I believe this issue happens because behaviour is different from when optimization in in1d happens or not.
If optimization happens, the comparison becomes np.array([0]) == '0' which is False.
If optimization does not happen, the type is silently converted to string while doing np.concatenate((ar1, ar2)), and all values are treated as string. Therefore it returns True.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in numpy/lib/arraysetops.py around the referenced lines 575-584 and 591, then run the two np.isin examples from the issue to compare the optimized and non-optimized paths. Determine whether the inconsistent result should be fixed or documented; done means the behavior is consistent or its conditions and limitations are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100