pydata / pydata/xarray

Merge fails when sparse Dataset has overlapping dimension values

Open
#3,445 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-combine
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Sparse numpy arrays used in a merge operation seem to fail under certain coordinate settings. for example, this works perfectly:

import xarray as xr
import numpy as np

data_array1 = xr.DataArray(data,name='default',
                           dims=['source','receiver','time'],
                          coords={'source':['X.1'],
                                  'receiver':['X.2'],
                                  'time':time}).to_dataset()
data_array2 = xr.DataArray(data,name='default',
                           dims=['source','receiver','time'],
                          coords={'source':['X.2'],
                                  'receiver':['X.1'],
                                  'time':time}).to_dataset()

dataset1 = xr.merge([data_array1,data_array2])

But this raises an IndexError: Only indices with at most one iterable index are supported. from the sparse package:

import xarray as xr
import numpy as np
import sparse

data = sparse.COO.from_numpy(np.random.uniform(-1,1,(1,1,100)))
time = np.linspace(0,1,num=100)

data_array1 = xr.DataArray(data,name='default',
                           dims=['source','receiver','time'],
                          coords={'source':['X.1'],
                                  'receiver':['X.2'],
                                  'time':time}).to_dataset()
data_array2 = xr.DataArray(data,name='default',
                           dims=['source','receiver','time'],
                          coords={'source':['X.2'],
                                  'receiver':['X.1'],
                                  'time':time}).to_dataset()

dataset1 = xr.merge([data_array1,data_array2])

I have noticed this occurs when the merger would seem to add dimensions filled with nan values.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the sparse.COO reproducer in the issue and compare it with the dense NumPy case. Trace the xarray merge path where overlapping coordinates introduce dimensions filled with NaN values, then verify that merging the sparse datasets completes without the sparse indexing error and preserves the expected dimensions.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.