Merging into existing dataset and saving to netcdf throws KeyError
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened:
I am trying to write a program which adds data to an existing Dataset saved as a netcdf file. Saving a Dataset to a netcdf file, then loading it back and merging it with another Datset works fine. However, when trying to save this combined Dataset to the same filename the program throws KeyError. If another filename is used or if there is no overlap between the dimensions everything works.
What you expected to happen:
I expected the Dataset to overwrite the existing file.
Minimal Complete Verifiable Example:
# Using xarray version 0.16.0
from xarray import DataArray, merge, open_dataset
from numpy import zeros
# First DataArray
da_a = DataArray(zeros((1,1)),coords=[('x',[1]),('a',[2])])
ds_a = da_a.to_dataset(name="var_from_da_a")
# Second DataArray
da_b = DataArray(zeros((1,1)),coords=[('x',[1]),('b',[3])])
ds_b = da_b.to_dataset(name="var_from_da_b")
# This works
ds = merge([ds_a,ds_b])
ds.to_netcdf('works.nc')
# This fails, throws KeyError: 'var_from_da_a'
ds_a.to_netcdf('fails.nc')
with open_dataset('fails.nc') as handle:
ds_alt = merge([handle,ds_b])
ds_alt.to_netcdf('fails.nc')
Anything else we need to know?:
Copying the handle or manually trying to close the old file (although that should be handled by the "with" statement) does not help.
Also it's my first time opening an issue for xarray so I'm sorry if it's not appropriate and in general I'm really impressed with xarray!
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None
libhdf5: 1.10.5
libnetcdf: 4.6.3
xarray: 0.16.0
pandas: 1.0.5
numpy: 1.16.4
scipy: 1.3.0
netCDF4: 1.5.1.2
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: None
cftime: 1.0.3.4
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.18.0
distributed: 2.18.0
matplotlib: 3.1.1
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 49.2.0
pip: 20.1.1
conda: None
pytest: None
IPython: 7.6.1
sphinx: 2.1.2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the supplied minimal example with xarray 0.16.0, then trace the interaction between open_dataset, merge, and to_netcdf when the output filename is reused. Add regression coverage for overlapping dimensions and verify that writing the merged Dataset to the existing filename no longer raises KeyError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100