Dropping of unaligned Data at assignment to Dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened:
I recently ran into the trouble as I assigned data generate by an external program to a dataset, and suddenly the dataset contained only NaN, see the example below. The issue was, that the program rounded numbers to 10 digits, so the coordinates didn't match anymore. xarray silently ignores this.
What you expected to happen:
I would have expected an error or at least a warning, when the coordinates don't match.
The current behavior can lead to bugs which are very hard to trace.
Minimal Complete Verifiable Example:
import numpy as np
import xarray as xr
x = np.linspace(0, 1)
dataset = xr.Dataset(coords={'x': x})
data = xr.DataArray(np.random.random(50), dims=['x'], coords={'x': np.around(x, decimals=10)})
dataset['data'] = data
print(dataset.data)
print(dataset.coords['x'])
print(data.coords['x'])
Output:
# print(dataset.data)
<xarray.DataArray 'data' (x: 50)>
array([0.20134419, nan, nan, nan, nan,
nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan,
nan, nan, nan, nan, 0.98357925])
Coordinates:
* x (x) float64 0.0 0.02041 0.04082 0.06122 ... 0.9592 0.9796 1.0
# print(dataset.coords['x'])
<xarray.DataArray 'x' (x: 50)>
array([0. , 0.020408, 0.040816, 0.061224, 0.081633, 0.102041, 0.122449,
0.142857, 0.163265, 0.183673, 0.204082, 0.22449 , 0.244898, 0.265306,
0.285714, 0.306122, 0.326531, 0.346939, 0.367347, 0.387755, 0.408163,
0.428571, 0.44898 , 0.469388, 0.489796, 0.510204, 0.530612, 0.55102 ,
0.571429, 0.591837, 0.612245, 0.632653, 0.653061, 0.673469, 0.693878,
0.714286, 0.734694, 0.755102, 0.77551 , 0.795918, 0.816327, 0.836735,
0.857143, 0.877551, 0.897959, 0.918367, 0.938776, 0.959184, 0.979592,
1. ])
Coordinates:
* x (x) float64 0.0 0.02041 0.04082 0.06122 ... 0.9592 0.9796 1.0
# print(data.coords['x'])
<xarray.DataArray 'x' (x: 50)>
array([0. , 0.020408, 0.040816, 0.061224, 0.081633, 0.102041, 0.122449,
0.142857, 0.163265, 0.183673, 0.204082, 0.22449 , 0.244898, 0.265306,
0.285714, 0.306122, 0.326531, 0.346939, 0.367347, 0.387755, 0.408163,
0.428571, 0.44898 , 0.469388, 0.489796, 0.510204, 0.530612, 0.55102 ,
0.571429, 0.591837, 0.612245, 0.632653, 0.653061, 0.673469, 0.693878,
0.714286, 0.734694, 0.755102, 0.77551 , 0.795918, 0.816327, 0.836735,
0.857143, 0.877551, 0.897959, 0.918367, 0.938776, 0.959184, 0.979592,
1. ])
Coordinates:
* x (x) float64 0.0 0.02041 0.04082 0.06122 ... 0.9592 0.9796 1.0
Anything else we need to know?:
Environment:
Output of xr.show_versions()
$ py -c "import xarray as xr; xr.show_versions()"
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 4.15.0-118-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.2
libnetcdf: 4.6.1
xarray: 0.16.1
pandas: 1.0.5
numpy: 1.18.5
scipy: 1.5.0
netCDF4: 1.4.2
pydap: None
h5netcdf: 0.8.1
h5py: 2.9.0
Nio: None
zarr: None
cftime: 1.1.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.13.0
distributed: 2.13.0
matplotlib: 3.2.1
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 46.1.3
pip: 19.3.1
conda: 4.8.5
pytest: 5.1.2
IPython: 7.18.1
sphinx: 3.0.2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the copy-pastable Python example and tracing Dataset assignment and coordinate alignment. Add a regression test for rounded, nonmatching coordinates, then establish whether completion should produce a warning or an error; the mismatch must no longer be silent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100