pydata / pydata/xarray

xr.combine_nested() fails when passed nested DataSets

Open
#3,315 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-combine
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

xr.__version__ '0.13.0'

xr.combine_nested() works when passed a nested list of DataArray objects.

da1 = xr.DataArray(name="a", data=[[0]], dims=["x", "y"])
da2 = xr.DataArray(name="b", data=[[1]], dims=["x", "y"])
da3 = xr.DataArray(name="a", data=[[2]], dims=["x", "y"])
da4 = xr.DataArray(name="b", data=[[3]], dims=["x", "y"])
xr.combine_nested([[da1, da2], [da3, da4]], concat_dim=["x", "y"])

returns

<xarray.DataArray 'a' (x: 2, y: 2)>
array([[0, 1],
       [2, 3]])
Dimensions without coordinates: x, y

but fails if passed a nested list of DataSet objects.

ds1 = da1.to_dataset()
ds2 = da2.to_dataset()
ds3 = da3.to_dataset()
ds4 = da4.to_dataset()
xr.combine_nested([[ds1, ds2], [ds3, ds4]], concat_dim=["x", "y"])

returns

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-8-c0035883fc68> in <module>
      3 ds3 = da3.to_dataset()
      4 ds4 = da4.to_dataset()
----> 5 xr.combine_nested([[ds1, ds2], [ds3, ds4]], concat_dim=["x", "y"])

~/repos/contribute/xarray/xarray/core/combine.py in combine_nested(datasets, concat_dim, compat, data_vars, coords, fill_value, join)
    462         ids=False,
    463         fill_value=fill_value,
--> 464         join=join,
    465     )
    466 

~/repos/contribute/xarray/xarray/core/combine.py in _nested_combine(datasets, concat_dims, compat, data_vars, coords, ids, fill_value, join)
    305         coords=coords,
    306         fill_value=fill_value,
--> 307         join=join,
    308     )
    309     return combined

~/repos/contribute/xarray/xarray/core/combine.py in _combine_nd(combined_ids, concat_dims, data_vars, coords, compat, fill_value, join)
    196             compat=compat,
    197             fill_value=fill_value,
--> 198             join=join,
    199         )
    200     (combined_ds,) = combined_ids.values()

~/repos/contribute/xarray/xarray/core/combine.py in _combine_all_along_first_dim(combined_ids, dim, data_vars, coords, compat, fill_value, join)
    218         datasets = combined_ids.values()
    219         new_combined_ids[new_id] = _combine_1d(
--> 220             datasets, dim, compat, data_vars, coords, fill_value, join
    221         )
    222     return new_combined_ids

~/repos/contribute/xarray/xarray/core/combine.py in _combine_1d(datasets, concat_dim, compat, data_vars, coords, fill_value, join)
    246                 compat=compat,
    247                 fill_value=fill_value,
--> 248                 join=join,
    249             )
    250         except ValueError as err:

~/repos/contribute/xarray/xarray/core/concat.py in concat(objs, dim, data_vars, coords, compat, positions, fill_value, join)
    131             "objects, got %s" % type(first_obj)
    132         )
--> 133     return f(objs, dim, data_vars, coords, compat, positions, fill_value, join)
    134 
    135 

~/repos/contribute/xarray/xarray/core/concat.py in _dataset_concat(datasets, dim, data_vars, coords, compat, positions, fill_value, join)
    363     for k in datasets[0].variables:
    364         if k in concat_over:
--> 365             vars = ensure_common_dims([ds.variables[k] for ds in datasets])
    366             combined = concat_vars(vars, dim, positions)
    367             assert isinstance(combined, Variable)

~/repos/contribute/xarray/xarray/core/concat.py in <listcomp>(.0)
    363     for k in datasets[0].variables:
    364         if k in concat_over:
--> 365             vars = ensure_common_dims([ds.variables[k] for ds in datasets])
    366             combined = concat_vars(vars, dim, positions)
    367             assert isinstance(combined, Variable)

~/repos/contribute/xarray/xarray/core/utils.py in __getitem__(self, key)
    383 
    384     def __getitem__(self, key: K) -> V:
--> 385         return self.mapping[key]
    386 
    387     def __iter__(self) -> Iterator[K]:

KeyError: 'a'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the nested DataSet example from the issue, then start in xarray/core/combine.py and xarray/core/concat.py, following the _combine_1d and _dataset_concat paths shown in the traceback. Add a regression test for the nested DataSet case and verify that combine_nested produces the same combined dimensions and values as the DataArray example.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.