pydata / pydata/xarray

Opendap access problem when subsetting via latitude and longitude...

Open
#4,052 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

I am trying to access a subset of a netcdf files hosted on a THREDDS server. I can inspect the metadata, but I cannot subset the file via lat and lon slices.
A download via the http link provided on the page works...

MCVE Code Sample
import xarray as xr

# this works
test1 = xr.open_dataset(URL)
display(test1)

# this also works
test2 = xr.open_dataset(URL).sel(lat=slice(30,40))
display(test2)

# this also works
test3 = xr.open_dataset(URL).sel(lon=slice(100,110))
display(test3)

# this fails
test4 = xr.open_dataset(URL).sel(lat=slice(30,40), lon=slice(100,110))
display(test4)

Problem Description

Error:

~/.pyenv/versions/miniconda3-latest/envs/datascience/lib/python3.7/site-packages/xarray/backends/common.py in robust_getitem(array, key, catch, max_retries, initial_delay)
     52     for n in range(max_retries + 1):
     53         try:
---> 54             return array[key]
     55         except catch:
     56             if n == max_retries:

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: Access failure

I also tried to use the pydap engine, but I'm not sure if this tells me something about the problem or if I use this option incorrectly...


URL = "https://thredds.daac.ornl.gov/thredds/dodsC/ornldaac/1247/T_CLAY.nc4"
test1 = xr.open_dataset(URL, engine='pydap')
test1

result:

UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128)
Versions
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.4 (default, Aug 13 2019, 15:17:50) 
[Clang 4.0.1 (tags/RELEASE_401/final)]
python-bits: 64
OS: Darwin
OS-release: 19.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.1

xarray: 0.15.1
pandas: 1.0.1
numpy: 1.18.1
scipy: 1.4.1
netCDF4: 1.4.2
pydap: installed
h5netcdf: None
h5py: None
Nio: None
zarr: 2.4.0
cftime: 1.0.4.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.0.21
cfgrib: None
iris: None
bottleneck: None
dask: 2.11.0
distributed: 2.11.0
matplotlib: 3.1.3
cartopy: 0.17.0
seaborn: 0.10.0
numbagg: None
setuptools: 45.2.0.post20200210
pip: 20.0.2
conda: None
pytest: None
IPython: 7.12.0
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the MCVE with xarray.open_dataset and the latitude/longitude .sel call against the supplied THREDDS URL. Read xarray/backends/common.py at robust_getitem and compare the netCDF4 and pydap paths shown in the report. Done means the remote subset behavior is understood and the issue is either fixed with coverage or documented as an external server/backend limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.