Issues with np.unique when dtype="object"
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
There are issues with numpy.unique when an array has dtype "object"
- when there are incomparable types in the array, an exception is raised
- when the types are all numeric, but the array contains
NaN, the result is incorrect.
Both of those issues seem to be caused by issues with sort.
Are unique and sort known to not work when the dtype is "object"? I didn't see any indication in the documentation for either function.
Reproducing code example:
Here, np.unique works as expected:
In [1]: import numpy as np
In [2]: arr = np.array([0.0, 1, np.nan, 1.0, 0])
In [3]: np.sort(arr)
Out[3]: array([ 0., 0., 1., 1., nan])
In [4]: np.unique(arr)
Out[4]: array([ 0., 1., nan])
Here, mixed type inside of dtype="object" causes an exception:
In [5]: arr = np.array(["a", "b", 1.0, "b", "a"], dtype="object")
In [6]: np.sort(arr)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-6-3b220bbb055a> in <module>
----> 1 np.sort(arr)
<__array_function__ internals> in sort(*args, **kwargs)
~/anaconda3/envs/tabnet/lib/python3.8/site-packages/numpy/core/fromnumeric.py in sort(a, axis, kind, order)
989 else:
990 a = asanyarray(a).copy(order="K")
--> 991 a.sort(axis=axis, kind=kind, order=order)
992 return a
993
TypeError: '<' not supported between instances of 'float' and 'str'
Here, no exception is raised, but the results are incorrect:
In [7]: arr = np.array([0.0, 1, np.nan, 1.0, 0], dtype="object")
In [8]: np.sort(arr)
Out[8]: array([0.0, 1, nan, 0, 1.0], dtype=object)
In [9]: np.unique(arr)
Out[9]: array([0.0, 1, nan, 0, 1.0], dtype=object)
In [10]: arr = np.array([0.0, 1.0, np.nan, 1.0, 0.0], dtype="object")
In [11]: np.sort(arr)
Out[11]: array([0.0, 1.0, nan, 0.0, 1.0], dtype=object)
In [12]: np.unique(arr)
Out[12]: array([0.0, 1.0, nan, 0.0, 1.0], dtype=object)
The documentation for sort says "In numpy versions >= 1.4.0 nan values are sorted to the end."
NumPy/Python version information:
Seen in NumPy versions 1.19.5, Python version 3.8.5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the issue's examples through np.sort and np.unique to reproduce the mixed-type exception and the object-array NaN results. Read the implementations and documentation for both entry points, then clarify the intended behavior before defining completion; the issue does not identify a target fix or test file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100