pandas-dev / pandas-dev/pandas

BUG: drop_duplicates() doesn't work for object dtype series containing numpy nans

Open
#16,632 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug duplicated Missing-data
Dominant language
Python
Stars
49.8k
Forks
20.4k
Avg merge
1d 16h
Merged PRs (30d)
322

Description

Code Sample, a copy-pastable example if possible
import numpy as np
import pandas as pd
import pandas.testing as pdt

s1 = pd.Series([np.nan, np.nan, 'text'])
s2 = pd.Series([np.float64(np.nan), np.float64(np.nan),'text'])

# This doesn't blow up, thinks s1 and s2 are the same
pdt.assert_series_equal(s1, s2)

s1_unique = s1.drop_duplicates()
s2_unique = s2.drop_duplicates()

# This blows up
pdt.assert_series_equal(s1_unique, s2_unique)

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-415-6908d74400cd> in <module>()
----> 1 pdt.assert_series_equal(s1_unique, s2_unique)

~/local/ts3/lib/python3.6/site-packages/pandas/util/testing.py in assert_series_equal(left, right, check_dtype, check_index_type, check_series_type, check_less_precise, check_names, check_exact, check_datetimelike_compat, check_categorical, obj)
   1276         raise_assert_detail(obj, 'Series length are different',
   1277                             '{0}, {1}'.format(len(left), left.index),
-> 1278                             '{0}, {1}'.format(len(right), right.index))
   1279 
   1280     # index comparison

~/local/ts3/lib/python3.6/site-packages/pandas/util/testing.py in raise_assert_detail(obj, message, left, right, diff)
   1147         msg = msg + "\n[diff]: {diff}".format(diff=diff)
   1148 
-> 1149     raise AssertionError(msg)
   1150 
   1151 

AssertionError: Series are different

Series length are different
[left]:  2, Int64Index([0, 2], dtype='int64')
[right]: 3, Int64Index([0, 1, 2], dtype='int64')

Problem description

When dealing with mixed dtype Series (sometimes as a result of .T followed by slice operation from dataframes), the drop_duplicates() call is very surprising, as it doesn't work for np.float64(np.nan). I would expect the htable.duplicated_object(values) call to also work with mixed dtypes containing np.float64 nan values.

The drop_duplicates() call does work for python's builtin float.nan, however.

Expected Output
import pandas as pd
import pandas.testing as pdt

s1 = pd.Series([np.nan, np.nan, 'text'])
s2 = pd.Series([np.float64(np.nan), np.float64(np.nan),'text'])

# This doesn't blow up, thinks s1 and s2 are the same
pdt.assert_series_equal(s1, s2)

s1_unique = s1.drop_duplicates()
s2_unique = s2.drop_duplicates()

# The following assertions should not blow up
assert len(s1_unique) == 2
assert len(s2_unique) == 2
pdt.assert_series_equal(s1_unique, s2_unique)
Output of pd.show_versions()
# Paste the output here pd.show_versions() here INSTALLED VERSIONS ------------------ commit: None python: 3.6.1.final.0 python-bits: 64 OS: Darwin OS-release: 16.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8

pandas: 0.20.2
pytest: 3.1.1
pip: 9.0.1
setuptools: 36.0.1
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
xarray: 0.9.5
IPython: 6.1.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: 1.5.1
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.8
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.8.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.10
pymysql: None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: 0.4.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the provided Series example, then inspect Series.drop_duplicates() and the htable.duplicated_object(values) path mentioned in the issue. Add a regression test covering object Series values containing repeated numpy.float64 NaNs; done means both Series produce two rows and compare equal.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.