numpy / numpy/numpy

Issues with np.unique when dtype="object"

Open
#18,147 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

There are issues with numpy.unique when an array has dtype "object"

  1. when there are incomparable types in the array, an exception is raised
  2. when the types are all numeric, but the array contains NaN, the result is incorrect.

Both of those issues seem to be caused by issues with sort.

Are unique and sort known to not work when the dtype is "object"? I didn't see any indication in the documentation for either function.

Reproducing code example:

Here, np.unique works as expected:

In [1]: import numpy as np

In [2]: arr = np.array([0.0, 1, np.nan, 1.0, 0])

In [3]: np.sort(arr)
Out[3]: array([ 0.,  0.,  1.,  1., nan])

In [4]: np.unique(arr)
Out[4]: array([ 0.,  1., nan])

Here, mixed type inside of dtype="object" causes an exception:

In [5]: arr = np.array(["a", "b", 1.0, "b", "a"], dtype="object")

In [6]: np.sort(arr)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-3b220bbb055a> in <module>
----> 1 np.sort(arr)

<__array_function__ internals> in sort(*args, **kwargs)

~/anaconda3/envs/tabnet/lib/python3.8/site-packages/numpy/core/fromnumeric.py in sort(a, axis, kind, order)
    989     else:
    990         a = asanyarray(a).copy(order="K")
--> 991     a.sort(axis=axis, kind=kind, order=order)
    992     return a
    993 

TypeError: '<' not supported between instances of 'float' and 'str'

Here, no exception is raised, but the results are incorrect:

In [7]: arr = np.array([0.0, 1, np.nan, 1.0, 0], dtype="object")

In [8]: np.sort(arr)
Out[8]: array([0.0, 1, nan, 0, 1.0], dtype=object)

In [9]: np.unique(arr)
Out[9]: array([0.0, 1, nan, 0, 1.0], dtype=object)

In [10]: arr = np.array([0.0, 1.0, np.nan, 1.0, 0.0], dtype="object")

In [11]: np.sort(arr)
Out[11]: array([0.0, 1.0, nan, 0.0, 1.0], dtype=object)

In [12]: np.unique(arr)
Out[12]: array([0.0, 1.0, nan, 0.0, 1.0], dtype=object)

The documentation for sort says "In numpy versions >= 1.4.0 nan values are sorted to the end."

NumPy/Python version information:

Seen in NumPy versions 1.19.5, Python version 3.8.5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the issue's examples through np.sort and np.unique to reproduce the mixed-type exception and the object-array NaN results. Read the implementations and documentation for both entry points, then clarify the intended behavior before defining completion; the issue does not identify a target fix or test file.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.