Matrix Index is tilted using combine_by_coords
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
My calculations return a strange tilted index. Why does this happen?
What happened:
I combined several user defined chunked netcdf data (900 chunks) into one dataset. For this I used the default combine_by_coords in ds.open_mf_dataset(). My result was a tilted grid index - upper left corner i=0, j=1167.
Beforehand I calculated some indices on these chunks and combined them with the default combine='by_coords' in ds.open_mf_dataset() but also tested the combine='nested' separately.
The ones where I used default combine='by_coords' for all functions returned the tilted index.
The ones where I used combine='nested' beforehand and then default combine='by_coords' returned the correct index.
What you expected to happen:
No tilted index.
Minimal Complete Verifiable Example:
##returning wrong index
#calculating some climatic indices on numbered chunks and combined them to one ds per chunk
with xr.open_mfdataset(pathtofile+'annual*'+chunknumber+'.nc', chunks=-1, parallel=True, engine='h5netcdf') as ds:
ds.to_netcdf(pathtofile, format="NETCDF4_CLASSIC", engine="netcdf4")
##combining all
with xr.open_mfdataset(pathtofile+'climateAnnual*.nc', chunks=-1, parallel=True, engine='h5netcd') as ds:
ds.to_netcdf(pathtofile, format="NETCDF4_CLASSIC", engine="netcdf4")
##################################################
##returning correct index
#calculating some climatic indices on numbered chunks and combined them to one ds per chunk
with xr.open_mfdataset(pathtofile+'annual*'+chunknumber+'.nc', chunks=-1, parallel=True, engine='h5netcdf', combine='nested') as ds:
ds.to_netcdf(pathtofile, format="NETCDF4_CLASSIC", engine="netcdf4")
##combining all
with xr.open_mfdataset(pathtofile+'climateAnnual*.nc', chunks=-1, parallel=True, engine='h5netcd') as ds:
ds.to_netcdf(pathtofile, format="NETCDF4_CLASSIC", engine="netcdf4")
Anything else we need to know?:
Environment:
Python 3.7.4
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.7.4 (default, Jun 3 2020, 14:52:58)
[GCC 8.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.15.2.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8
LOCALE: ('de_DE', 'UTF-8')
libhdf5: 1.12.0
libnetcdf: 4.7.4
xarray: 0.18.2
pandas: 0.25.3
numpy: 1.17.3
scipy: 1.3.1
netCDF4: 1.5.7
pydap: installed
h5netcdf: 0.11.0
h5py: 3.3.0
Nio: None
zarr: 2.8.3
cftime: 1.5.0
nc_time_axis: 1.3.1
PseudoNetCDF: None
rasterio: 1.2.6
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2021.06.2
distributed: 2021.06.2
matplotlib: 3.4.2
cartopy: None
seaborn: 0.11.1
numbagg: 0.2.1
pint: 0.17
setuptools: 57.0.0
pip: 21.1.3
conda: None
pytest: None
IPython: None
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with xarray.open_mfdataset, especially the default combine_by_coords path, and reproduce the result using the provided chunked NetCDF workflow. Compare it with combine='nested' and inspect how the coordinates and indices are ordered across the 900 chunks. Done means the cause of the tilted grid index is identified and the behavior is corrected or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100