bug with to_netcdf when reading dataset via THREDDS
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
I am trying to extract data for a specific station from data observations accessed via THREDDS. At first I got an error but a colleague found a way to make it work. In the code below it works if APPLY_PATCH=True but it crashes if APPLY_PATCH=False.
I do not know if this behaviour is expected and/or linked to the way the data is 'served' via THREDDS but I would never have found the way around it by myself.
import xarray as xr
CLEF_STATION = "MONTREAL"
NOM_VAR = 'prsn'
APPLY_PATCH = False
url_pavics = "http://pavics.ouranos.ca:8083/twitcher/ows/proxy/thredds/dodsC"
url_donnees_homo = "/".join((url_pavics, "datasets/station_obs/ECCC_AHCCD_gen2_precipitation.ncml"))
ds = xr.open_dataset(url_donnees_homo)
# data for specific station(s)
dsr = ds.sel(station=ds.station_name.str.contains(CLEF_STATION))
# apply patch
if APPLY_PATCH:
for var in dsr.variables:
if dsr[var].dtype == 'O':
dsr[var].encoding = {}
# save to a netcdf file
dsr.to_netcdf("out.nc", format="NETCDF4_CLASSIC")
What happened:
When running this I get the following error
runfile('/Users/sbiner/cloud/OneDrive - Ouranos inc/job/stagiaires_etudiants/2021_tempete_neige/code/issue_xaray.py', wdir='/Users/sbiner/cloud/OneDrive - Ouranos inc/job/stagiaires_etudiants/2021_tempete_neige/code')
Traceback (most recent call last):
File "src/netCDF4/_netCDF4.pyx", line 4931, in netCDF4._netCDF4.Variable.__setitem__
ValueError: cannot reshape array of size 7 into shape (1,1)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/sbiner/cloud/OneDrive - Ouranos inc/job/stagiaires_etudiants/2021_tempete_neige/code/issue_xaray.py", line 14, in <module>
dsr.to_netcdf("out.nc", format="NETCDF4_CLASSIC")
File "/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/core/dataset.py", line 1689, in to_netcdf
return to_netcdf(
File "/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/api.py", line 1107, in to_netcdf
dump_to_store(
File "/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/api.py", line 1154, in dump_to_store
store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
File "/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/common.py", line 255, in store
self.set_variables(
File "/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/common.py", line 297, in set_variables
writer.add(source, target)
File "/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/common.py", line 146, in add
target[...] = source
File "/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 64, in __setitem__
data[key] = value
File "src/netCDF4/_netCDF4.pyx", line 4933, in netCDF4._netCDF4.Variable.__setitem__
File "<__array_function__ internals>", line 5, in broadcast_to
File "/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/numpy/lib/stride_tricks.py", line 411, in broadcast_to
return _broadcast_to(array, shape, subok=subok, readonly=True)
File "/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/numpy/lib/stride_tricks.py", line 348, in _broadcast_to
it = np.nditer(
ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (1,7) and requested shape (1,1)
What you expected to happen:
I expect a netcdf file to be written
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.8.8 (default, Apr 13 2021, 12:59:45)
[Clang 10.0.0 ]
python-bits: 64
OS: Darwin
OS-release: 19.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: fr_CA.UTF-8
LANG: fr_CA.UTF-8
LOCALE: fr_CA.UTF-8
libhdf5: 1.10.6
libnetcdf: 4.6.1
xarray: 0.17.0
pandas: 1.2.4
numpy: 1.20.1
scipy: 1.6.2
netCDF4: 1.5.6
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.4.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2021.04.0
distributed: 2021.04.0
matplotlib: 3.3.4
cartopy: 0.18.0
seaborn: 0.11.1
numbagg: None
pint: None
setuptools: 52.0.0.post20210125
pip: 20.3.3
conda: None
pytest: None
IPython: 7.22.0
sphinx: 4.0.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Dataset.to_netcdf and the backend paths shown in the traceback, especially xarray/core/backends/common.py and xarray/backends/netCDF4_.py. Reproduce the THREDDS selection and write operation, then identify why the object-valued variables produce a shape mismatch. Done means the provided dataset can be written without applying the workaround and a regression test covers the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100