pydata / pydata/xarray

dataset attrs list of strings to_netcdf() error

Open
#7,608 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-backends topic-error reporting usage question
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

when I try and save the dataset with an attribute as a list of strings it fails.

See example code


import xarray as xr
import pandas as pd
import numpy as np

ds = xr.Dataset()

temp = 15 + 8 * np.random.randn(2, 2, 3)

precip = 10 * np.random.rand(2, 2, 3)

lon = [[-99.83, -99.32], [-99.79, -99.23]]

lat = [[42.25, 42.21], [42.63, 42.59]]

ds["temperature"] = (("x", "y", "time"), temp)

ds["temperature_double"] = (("x", "y", "time"), temp * 2)

ds["precipitation"] = (("x", "y", "time"), precip)

ds.coords["lat"] = (("x", "y"), lat)

ds.coords["lon"] = (("x", "y"), lon)

ds.coords["time"] = pd.date_range("2014-09-06", periods=3)

ds.coords["reference_time"] = pd.Timestamp("2014-09-05")

ds.attrs['test'] = [1,2,3]
ds.to_netcdf('test.nc')
# no error

ds.attrs['test'] = ['1','2','3']
ds.to_netcdf('test.nc')
  File "C:\Users\xxxx\Anaconda3\envs\base1\lib\site-packages\scipy\io\_netcdf.py", line 555, in _write_att_values
   nc_type = REVERSE[values.dtype.char, values.dtype.itemsize]

 KeyError: ('U', 4)
What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output

No response

Anything else we need to know?

No response

Environment

xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.10.9 | packaged by conda-forge | (main, Jan 11 2023, 15:15:40) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: ('English_United States', '1252')
libhdf5: 1.10.6
libnetcdf: None

xarray: 2022.11.0
pandas: 1.5.2
numpy: 1.21.5
scipy: 1.9.3
netCDF4: None
pydap: None
h5netcdf: None
h5py: 3.7.0
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.3.5
dask: 2022.05.0
distributed: 2022.5.0
matplotlib: 3.6.2
cartopy: None
seaborn: 0.12.2
numbagg: None
fsspec: 2022.11.0
cupy: None
pint: 0.20.1
sparse: None
flox: None
numpy_groupies: None
setuptools: 65.6.3
pip: 22.3.1
conda: None
pytest: 7.1.2
IPython: 7.31.1
sphinx: 5.0.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Dataset.to_netcdf() and the scipy/io/_netcdf.py _write_att_values traceback shown in the report. Reproduce the two ds.attrs['test'] assignments and compare the successful numeric-list write with the failing string-list write. Done means the string-list attribute case writes successfully, with a regression test covering it.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.