pydata / pydata/xarray

Dropping of unaligned Data at assignment to Dataset

Open
#4,507 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:
I recently ran into the trouble as I assigned data generate by an external program to a dataset, and suddenly the dataset contained only NaN, see the example below. The issue was, that the program rounded numbers to 10 digits, so the coordinates didn't match anymore. xarray silently ignores this.

What you expected to happen:
I would have expected an error or at least a warning, when the coordinates don't match.
The current behavior can lead to bugs which are very hard to trace.

Minimal Complete Verifiable Example:

import numpy as np
import xarray as xr

x = np.linspace(0, 1)
dataset = xr.Dataset(coords={'x': x})
data = xr.DataArray(np.random.random(50), dims=['x'], coords={'x': np.around(x, decimals=10)})

dataset['data'] = data
print(dataset.data)
print(dataset.coords['x'])
print(data.coords['x'])

Output:

# print(dataset.data)
<xarray.DataArray 'data' (x: 50)>
array([0.20134419,        nan,        nan,        nan,        nan,
              nan,        nan,        nan,        nan,        nan,
              nan,        nan,        nan,        nan,        nan,
              nan,        nan,        nan,        nan,        nan,
              nan,        nan,        nan,        nan,        nan,
              nan,        nan,        nan,        nan,        nan,
              nan,        nan,        nan,        nan,        nan,
              nan,        nan,        nan,        nan,        nan,
              nan,        nan,        nan,        nan,        nan,
              nan,        nan,        nan,        nan, 0.98357925])
Coordinates:
  * x        (x) float64 0.0 0.02041 0.04082 0.06122 ... 0.9592 0.9796 1.0
# print(dataset.coords['x'])
<xarray.DataArray 'x' (x: 50)>
array([0.      , 0.020408, 0.040816, 0.061224, 0.081633, 0.102041, 0.122449,
       0.142857, 0.163265, 0.183673, 0.204082, 0.22449 , 0.244898, 0.265306,
       0.285714, 0.306122, 0.326531, 0.346939, 0.367347, 0.387755, 0.408163,
       0.428571, 0.44898 , 0.469388, 0.489796, 0.510204, 0.530612, 0.55102 ,
       0.571429, 0.591837, 0.612245, 0.632653, 0.653061, 0.673469, 0.693878,
       0.714286, 0.734694, 0.755102, 0.77551 , 0.795918, 0.816327, 0.836735,
       0.857143, 0.877551, 0.897959, 0.918367, 0.938776, 0.959184, 0.979592,
       1.      ])
Coordinates:
  * x        (x) float64 0.0 0.02041 0.04082 0.06122 ... 0.9592 0.9796 1.0
# print(data.coords['x'])
<xarray.DataArray 'x' (x: 50)>
array([0.      , 0.020408, 0.040816, 0.061224, 0.081633, 0.102041, 0.122449,
       0.142857, 0.163265, 0.183673, 0.204082, 0.22449 , 0.244898, 0.265306,
       0.285714, 0.306122, 0.326531, 0.346939, 0.367347, 0.387755, 0.408163,
       0.428571, 0.44898 , 0.469388, 0.489796, 0.510204, 0.530612, 0.55102 ,
       0.571429, 0.591837, 0.612245, 0.632653, 0.653061, 0.673469, 0.693878,
       0.714286, 0.734694, 0.755102, 0.77551 , 0.795918, 0.816327, 0.836735,
       0.857143, 0.877551, 0.897959, 0.918367, 0.938776, 0.959184, 0.979592,
       1.      ])
Coordinates:
  * x        (x) float64 0.0 0.02041 0.04082 0.06122 ... 0.9592 0.9796 1.0

Anything else we need to know?:

Environment:

Output of xr.show_versions()
$ py -c "import xarray as xr; xr.show_versions()"
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 4.15.0-118-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.2
libnetcdf: 4.6.1

xarray: 0.16.1
pandas: 1.0.5
numpy: 1.18.5
scipy: 1.5.0
netCDF4: 1.4.2
pydap: None
h5netcdf: 0.8.1
h5py: 2.9.0
Nio: None
zarr: None
cftime: 1.1.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.13.0
distributed: 2.13.0
matplotlib: 3.2.1
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 46.1.3
pip: 19.3.1
conda: 4.8.5
pytest: 5.1.2
IPython: 7.18.1
sphinx: 3.0.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the copy-pastable Python example and tracing Dataset assignment and coordinate alignment. Add a regression test for rounded, nonmatching coordinates, then establish whether completion should produce a warning or an error; the mismatch must no longer be silent.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.