pydata / pydata/xarray

coordinates not removed for variable encoding during reset_coords

Open
#7,245 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

When calling reset_coords on a dataset that is loaded from disk, the coordinates are not removed from the encoding of the variable.

This means, that at save time they will be resaved as coordinates... annoying. (and erroneous)

What did you expect to happen?

No response

Minimal Complete Verifiable Example
import xarray as xr

dataset = xr.Dataset(
    data_vars={'images': (('y', 'x'),  np.zeros((10, 2)))},
    coords={'zar': 1}
)

dataset.to_netcdf('foo.nc', mode='w')

# %%
foo_loaded = xr.open_dataset('foo.nc')

foo_loaded_reset = foo_loaded.reset_coords()

# %%
assert 'zar' in foo_loaded.coords
assert 'zar' not in foo_loaded_reset.coords
assert 'zar' in foo_loaded_reset.data_vars
foo_loaded_reset.to_netcdf('bar.nc', mode='w')

# %% Now load the dataset

bar_loaded = xr.open_dataset('bar.nc')
assert 'zar' not in bar_loaded.coords, 'zar is erroneously a coordinate'

# %%
# This is the problem
assert 'zar' not in foo_loaded_reset.images.encoding['coordinates'].split(' '), "zar should not be in here"
MVCE confirmation
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output

No response

Anything else we need to know?
        for _, variable in obj._variables.items():
            coords_in_encoding = set(variable.encoding.get('coordinates', ' ').split(' '))
            variable.encoding['coordinates'] = ' '.join(coords_in_encoding - set(names))

suggested fix in dataset.py, reset_coords

https://github.com/pydata/xarray/blob/513ee34f16cc8f9250a72952e33bf9b4c95d33d1/xarray/core/dataset.py#L1734

Environment
INSTALLED VERSIONS
------------------
commit: None
python: 3.9.13 | packaged by Ramona Optics | (main, Aug 31 2022, 22:30:30) 
[GCC 10.4.0]
python-bits: 64
OS: Linux
OS-release: 5.15.0-50-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.8.1

xarray: 2022.10.0
pandas: 1.5.1
numpy: 1.23.4
scipy: 1.9.3
netCDF4: 1.6.1
pydap: None
h5netcdf: 1.0.2
h5py: 3.7.0
Nio: None
zarr: 2.13.3
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2022.10.0
distributed: 2022.10.0
matplotlib: 3.6.1
cartopy: None
seaborn: None
numbagg: None
fsspec: 2022.10.0
cupy: None
pint: 0.20.1
sparse: None
flox: None
numpy_groupies: None
setuptools: 65.5.0
pip: 22.3
conda: 22.9.0
pytest: 7.2.0
IPython: 7.33.0
sphinx: 5.3.0
/home/mark/mambaforge/envs/mcam_dev/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in xarray/core/dataset.py at reset_coords, then reproduce the provided netCDF round-trip example. Check the variable encoding after coordinates are reset; done means the removed coordinate is absent from the encoding and is not restored as a coordinate when the result is saved and reopened.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.