pydata / pydata/xarray

DataArray.drop_isel / .drop_sel with duplicated initial time stamp - InvalidIndexError

Open
#6,605 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-indexing
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

I have a DataArray ds with 7098 time steps in cftime format. One of these time steps is a duplicate (pos: 7087 & 7088).
I tried to drop the first instance using both ds.drop_isel(time=[7087]) and ds.drop_sel(ds.time[7087]) but get the error:
pandas.errors.InvalidIndexError: Reindexing only valid with uniquely valued Index objects.
The method works if I do not have the duplicate in it. Maybe this is the wrong method in the first place but it seemed to be the way to go. I am confused on how to get rid of the duplicate, and, if the method is the correct method, if this might be an issue.

ds
Out[247]: 
<xarray.DataArray 'LST_Night_CMG' (time: 3, y: 2, x: 2)>
array([[[14357., 14357.],
        [14342., 14358.]],
       [[14409., 14409.],
        [14435., 14388.]],
       [[14409., 14409.],
        [14435., 14388.]]])
Coordinates:
  * time         (time) object 2019-09-18 00:00:00 ... 2019-09-19 00:00:00
  * x            (x) float64 67.98 68.03
  * y            (y) float64 40.12 40.07
    spatial_ref  int32 0
Attributes:
    coordinates:   spatial_ref band
    scale_factor:  1.0
    add_offset:    0.0


ds.time
Out[249]: 
<xarray.DataArray 'time' (time: 3)>
array([cftime.DatetimeProlepticGregorian(2019, 9, 18, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeProlepticGregorian(2019, 9, 19, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeProlepticGregorian(2019, 9, 19, 0, 0, 0, 0, has_year_zero=True)],
      dtype=object)
Coordinates:
  * time         (time) object 2019-09-18 00:00:00 ... 2019-09-19 00:00:00
    spatial_ref  int32 0



ds.drop_isel(time=1)
Traceback (most recent call last):
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\IPython\core\interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-248-0f89b7589e90>", line 1, in <module>
    ds.drop_isel(time=1)
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\dataarray.py", line 2413, in drop_isel
    dataset = dataset.drop_isel(indexers=indexers, **indexers_kwargs)
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\dataset.py", line 4565, in drop_isel
    ds = ds.loc[dimension_index]
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\dataset.py", line 563, in __getitem__
    return self.dataset.sel(key)
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\dataset.py", line 2505, in sel
    self, indexers=indexers, method=method, tolerance=tolerance
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\coordinates.py", line 422, in remap_label_indexers
    obj, v_indexers, method=method, tolerance=tolerance
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\indexing.py", line 120, in remap_label_indexers
    idxr, new_idx = index.query(labels, method=method, tolerance=tolerance)
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\indexes.py", line 240, in query
    indexer = get_indexer_nd(self.index, label, method, tolerance)
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\indexes.py", line 142, in get_indexer_nd
    flat_indexer = index.get_indexer(flat_labels, method=method, tolerance=tolerance)
  File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\pandas\core\indexes\base.py", line 3442, in get_indexer
    raise InvalidIndexError(self._requires_unique_msg)
pandas.errors.InvalidIndexError: Reindexing only valid with uniquely valued Index objects

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the duplicated cftime coordinate, then trace drop_isel through xarray/core/dataarray.py and xarray/core/dataset.py into the indexing code shown in the traceback. Check existing tests for drop_isel or duplicate indexes and add coverage for removing one positional duplicate. Done means the reported operation behaves consistently or the supported workaround is documented by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.