pydata / pydata/xarray

Merging into existing dataset and saving to netcdf throws KeyError

Open
#4,280 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-backends
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:
I am trying to write a program which adds data to an existing Dataset saved as a netcdf file. Saving a Dataset to a netcdf file, then loading it back and merging it with another Datset works fine. However, when trying to save this combined Dataset to the same filename the program throws KeyError. If another filename is used or if there is no overlap between the dimensions everything works.

What you expected to happen:
I expected the Dataset to overwrite the existing file.

Minimal Complete Verifiable Example:

# Using xarray version 0.16.0
from xarray import DataArray, merge, open_dataset
from numpy import zeros

# First DataArray
da_a = DataArray(zeros((1,1)),coords=[('x',[1]),('a',[2])])
ds_a = da_a.to_dataset(name="var_from_da_a")

# Second DataArray
da_b = DataArray(zeros((1,1)),coords=[('x',[1]),('b',[3])])
ds_b = da_b.to_dataset(name="var_from_da_b")

# This works
ds = merge([ds_a,ds_b])
ds.to_netcdf('works.nc')

# This fails, throws KeyError: 'var_from_da_a'
ds_a.to_netcdf('fails.nc')
with open_dataset('fails.nc') as handle:
    ds_alt = merge([handle,ds_b])
ds_alt.to_netcdf('fails.nc')

Anything else we need to know?:
Copying the handle or manually trying to close the old file (although that should be handled by the "with" statement) does not help.

Also it's my first time opening an issue for xarray so I'm sorry if it's not appropriate and in general I'm really impressed with xarray!

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None
libhdf5: 1.10.5
libnetcdf: 4.6.3

xarray: 0.16.0
pandas: 1.0.5
numpy: 1.16.4
scipy: 1.3.0
netCDF4: 1.5.1.2
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: None
cftime: 1.0.3.4
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.18.0
distributed: 2.18.0
matplotlib: 3.1.1
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 49.2.0
pip: 20.1.1
conda: None
pytest: None
IPython: 7.6.1
sphinx: 2.1.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the supplied minimal example with xarray 0.16.0, then trace the interaction between open_dataset, merge, and to_netcdf when the output filename is reused. Add regression coverage for overlapping dimensions and verify that writing the merged Dataset to the existing filename no longer raises KeyError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.