pydata / pydata/xarray

Matrix Index is tilted using combine_by_coords

Open
#5,760 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-combine
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

My calculations return a strange tilted index. Why does this happen?

What happened:
I combined several user defined chunked netcdf data (900 chunks) into one dataset. For this I used the default combine_by_coords in ds.open_mf_dataset(). My result was a tilted grid index - upper left corner i=0, j=1167.

Beforehand I calculated some indices on these chunks and combined them with the default combine='by_coords' in ds.open_mf_dataset() but also tested the combine='nested' separately.

The ones where I used default combine='by_coords' for all functions returned the tilted index.
The ones where I used combine='nested' beforehand and then default combine='by_coords' returned the correct index.

What you expected to happen:
No tilted index.

Minimal Complete Verifiable Example:


##returning wrong index
#calculating some climatic indices on numbered chunks and combined them to one ds per chunk
with xr.open_mfdataset(pathtofile+'annual*'+chunknumber+'.nc', chunks=-1, parallel=True, engine='h5netcdf') as ds:
    ds.to_netcdf(pathtofile, format="NETCDF4_CLASSIC", engine="netcdf4")

##combining all
with xr.open_mfdataset(pathtofile+'climateAnnual*.nc', chunks=-1, parallel=True, engine='h5netcd') as ds: 
    ds.to_netcdf(pathtofile, format="NETCDF4_CLASSIC", engine="netcdf4")

##################################################

##returning correct index
#calculating some climatic indices on numbered chunks and combined them to one ds per chunk
with xr.open_mfdataset(pathtofile+'annual*'+chunknumber+'.nc', chunks=-1, parallel=True, engine='h5netcdf', combine='nested') as ds:
    ds.to_netcdf(pathtofile, format="NETCDF4_CLASSIC", engine="netcdf4")

##combining all
with xr.open_mfdataset(pathtofile+'climateAnnual*.nc', chunks=-1, parallel=True, engine='h5netcd') as ds: 
    ds.to_netcdf(pathtofile, format="NETCDF4_CLASSIC", engine="netcdf4")



Anything else we need to know?:

Environment:
Python 3.7.4

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.4 (default, Jun 3 2020, 14:52:58)
[GCC 8.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.15.2.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8
LOCALE: ('de_DE', 'UTF-8')
libhdf5: 1.12.0
libnetcdf: 4.7.4

xarray: 0.18.2
pandas: 0.25.3
numpy: 1.17.3
scipy: 1.3.1
netCDF4: 1.5.7
pydap: installed
h5netcdf: 0.11.0
h5py: 3.3.0
Nio: None
zarr: 2.8.3
cftime: 1.5.0
nc_time_axis: 1.3.1
PseudoNetCDF: None
rasterio: 1.2.6
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2021.06.2
distributed: 2021.06.2
matplotlib: 3.4.2
cartopy: None
seaborn: 0.11.1
numbagg: 0.2.1
pint: 0.17
setuptools: 57.0.0
pip: 21.1.3
conda: None
pytest: None
IPython: None
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with xarray.open_mfdataset, especially the default combine_by_coords path, and reproduce the result using the provided chunked NetCDF workflow. Compare it with combine='nested' and inspect how the coordinates and indices are ordered across the 900 chunks. Done means the cause of the tilted grid index is identified and the behavior is corrected or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.