Interpolation breaks (upgrade from 0.17.0 to 0.19.0)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
I am using xarray to sample a 3D field along 2D Lagrangian trajectories (output from Parcels). The trajectory data consists of several spatial variables (longitude, latitude, depth), stored along the dimensions obs (essentially a time index) and traj (trajectory index). My aim is to interpolate the 3D field along 2D Lagrangian trajectories and obtain along-trajectory depth profiles of the data.
The code I used:
# Obtain profiles along all trajectories at the 0th observation
# `EulerianField` has dimensions ('Z', 'YC', 'XC')
# `LagrangianTrajectories.lon.isel(obs=0)` has dimension (`traj`); same for `lat`
EulerianField.interp(XC=LagrangianTrajectories.isel(obs=0).lon,
YC=LagrangianTrajectories.isel(obs=0).lat,
method='linear')
What happens in xarray 0.17.0:
Everything runs fine and quickly. A dataArray of the field is returned with dimensions (Z, traj); a depth profile for each trajectory (at obs=0).
What happens in xarray 0.19.0:
I get the following error:
numpy.core._exceptions.MemoryError: Unable to allocate 179. GiB for an array with shape (49, 62590, 62590) and data type bool
Note that the length of Z is 49, while that of traj is 62590. I suspect that xarray somehow tries to return an array with dimensions (Z, YC, XC), with YC and XC now having length 62590.
Minimal Complete Verifiable Example:
I haven't been able to reduce the issue to an MCVE. When I subset the trajectory dataset using .isel(obs=0, traj=slice(0,10), the behavior is as expected again (returning a dataArray with dimensions (Z, traj). I can send the files I'm working with using WeTransfer for testing, but they're several GBs. I hope that the issue contains enough hints for determining the cause.
Anything else we need to know?:
The trajectory data contains some NaNs, but when there are NaNs in the subsetted (traj=slice(0,10)), everything still works fine, so I don't think that's the cause.
Environment:
Output of xr.show_versions()
Below is the environment with xarray 0.17.0. The only difference with xarray 0.19.0 is the xarray version; all other packages are the same.INSTALLED VERSIONS
commit: None
python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27)
[GCC 9.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.36.2.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None
libhdf5: 1.10.6
libnetcdf: 4.8.0
xarray: 0.17.0
pandas: 1.3.1
numpy: 1.21.1
scipy: 1.6.3
netCDF4: 1.5.7
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.8.3
cftime: 1.5.0
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2021.07.1
distributed: 2021.07.1
matplotlib: 3.4.2
cartopy: 0.19.0.post1
seaborn: 0.11.1
numbagg: None
pint: None
setuptools: 49.6.0.post20210108
pip: 21.2.1
conda: 4.10.3
pytest: None
IPython: 7.25.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the interpolation call, comparing behavior between xarray 0.17.0 and 0.19.0 with the reported dimensions: Z=49 and traj=62590. Reduce the trajectory input or obtain the referenced files to reproduce the memory allocation, then verify that interpolation returns dimensions (Z, traj) without attempting the 179 GiB array.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100