numpy / numpy/numpy

Wrong results of ma.isin() and ma.in1d() for masked arrays

Open
#19,877 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

00 - Bug component: numpy.ma
Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

For masked arrays, isin() and in1d() produce wrong results. In particular, the result is masked at wrong places. The issue was discussed initially on stackoverflow and it seems that the involved sorting does not handle the masked value correctly.

Reproducing code example:
import numpy.ma as ma
a = ma.MaskedArray([[1,2,3],[4,5,6]], [[True,False,False],[False,False,False]])
ta = ma.array([1,4,5])
ma.isin(a, ta)

The output of ma.isin() is wrong (see below). I use the following workaround:

import numpy as np
def ma_isin(array, comparison):
    return ma.MaskedArray(
        data=np.isin(array, comparison),
        mask=array.mask.copy())
Output and expected results:

These are the results from the code above

>>> a
masked_array(
  data=[[--, 2, 3],
        [4, 5, 6]],
  mask=[[ True, False, False],
        [False, False, False]],
  fill_value=999999)
>>> ta
masked_array(data=[1, 4, 5],
             mask=False,
       fill_value=999999)
>>> ma.isin(a, ta)
masked_array(
  data=[[False, False, False],
        [True, True, --]],
  mask=[[False, False, False],
        [False, False,  True]],
  fill_value=True)

The issue is mainly that the last element in the result is masked, and probably also that the first element is not masked. I would expect the following output:

>>> ma_isin(a, ta)
masked_array(
  data=[[--, False, False],
        [True, True, False]],
  mask=[[ True, False, False],
        [False, False, False]],
  fill_value=True)

in1d() shows an analogue issue.

NumPy/Python version information:

1.21.2 3.9.4 (default, Apr 9 2021, 16:34:09)
[GCC 7.3.0]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the masked-array reproducer against ma.isin() and the analogous ma.in1d() case, then trace their masked-array sorting and comparison entry points. Verify that masked input positions remain masked in the result and that unmasked values receive the correct membership results; add regression coverage for both functions.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.