Xarray drops certain coordinates without warning after multiplying datasets
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
files.zip
Hi, I really like Xarray because it is doing a lot of work me. However, maybe in this case it was overdoing it a bit.
When making a field average of some data (data.nc) I use the grid area as weights (gridarea.nc). However, after multiplying the data with the gridarea the grid changed without warning.
The gridarea is constructed using 'cdo gridarea other_data.nc gridarea.nc'. The 'other_data.nc' in this case is from the same climate model and grid as data.nc, though from different simulations (and possibly post-processing).
Below a reproducable example, with the nc files attached.
MCVE Code Sample
import xarray as xr
data = xr.open_dataset('data.nc')
gridarea = xr.open_dataset('gridarea.nc')
Then I check whether the grids are the same
print((data.lat == gridarea.lat).all())
print((data.lon == gridarea.lon).all())
<xarray.DataArray 'lat' ()>
array(True)
<xarray.DataArray 'lon' ()>
array(True)
Then I multiply the data with the weights
new_data = data * gridarea
print(data)
<xarray.Dataset>
Dimensions: (lat: 16, lon: 30, time: 1)
Coordinates:
* time (time) datetime64[ns] 1995-01-01T12:00:00
* lon (lon) float64 0.0 1.125 2.25 3.375 4.5 ... 29.25 30.38 31.5 32.62
* lat (lat) float64 71.21 70.09 68.97 67.85 ... 57.76 56.64 55.51 54.39
Data variables:
U10M (time, lat, lon) float32 ...
print(new_data)
<xarray.Dataset>
Dimensions: (lat: 12, lon: 30, time: 1)
Coordinates:
* lat (lat) float64 71.21 70.09 68.97 65.61 ... 58.88 57.76 56.64 55.51
* time (time) datetime64[ns] 1995-01-01T12:00:00
* lon (lon) float64 0.0 1.125 2.25 3.375 4.5 ... 29.25 30.38 31.5 32.62
Data variables:
*empty*
Note the different latitude for both datasets
Expected Output
The expected output is to get a dataset with the same grid as the original dataset, however, the latitude has now length 12 i.s.o. length 16
Problem Description
After digging a bit more into this problem it turns out the grids are not identical.
print(data.lat == gridarea.lat)
<xarray.DataArray 'lat' (lat: 12)>
array([ True, True, True, True, True, True, True, True, True,
True, True, True])
Coordinates:
* lat (lat) float64 71.21 70.09 68.97 65.61 ... 58.88 57.76 56.64 55.51
whilst
print(data.lat.values == gridarea.lat.values)
[ True True True False False True False True True True True True
True True True False]
So the 'problem' is that xarray identifies the values which are not identical (e.g. 67.84978441466984 i.s.o. 67.84978441466983) and throws them out. Hence my original check if the grids are the same returns 'True' because all the values not equal are already thrown out. For me this introduced a 'conclusion rewriting error', because I averaged the data straight after multiplying with the weights. Hence, I didn't catch it at first. I'm not sure if this is 'wanted behavior'. For me it isn't at least ;-)
Could it be possible to either
- make xarray not throw out the unequal values when comparing data, or
- get a warning when you multiply two fields with the same dimensions (e.g ['time','lat','lon] and ['lat','lon'] and then it returns a different sized ['lat','lon']?
Output of xr.show_versions()
xarray: 0.14.0
pandas: 0.25.2
numpy: 1.17.3
scipy: 1.3.1
netCDF4: 1.4.0
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.0.3.4
nc_time_axis: 1.1.0
PseudoNetCDF: None
rasterio: 1.0.26
cfgrib: None
iris: None
bottleneck: 1.2.1
dask: 2.3.0
distributed: None
matplotlib: 3.1.1
cartopy: 0.17.0
seaborn: 0.9.0
numbagg: None
setuptools: 41.2.0
pip: 9.0.1
conda: None
pytest: None
IPython: 7.7.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the coordinate-alignment behavior using data.nc and gridarea.nc from files.zip, starting with the shown data * gridarea operation and the coordinate comparisons. Inspect the alignment path involved and determine how the reported mismatch should be surfaced. Done means the reproduced case no longer silently produces the unexpected 12-element latitude result, with coverage for the demonstrated inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100