pydata / pydata/xarray

Support for attributes with different dtypes when serialising to netcdf4

Open
#1,500 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-backends
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

At the moment, bool and dates aren't supported as attributes when serializing to netcdf4:

>>> da = xr.DataArray(range(5), attrs={'test': True})
>>> da 
<xarray.DataArray (dim_0: 5)>
array([0, 1, 2, 3, 4])
Dimensions without coordinates: dim_0
Attributes:
    test:     True

>>> da.to_netcdf('test_bool.nc')
...
TypeError: illegal data type for attribute, must be one of dict_keys(['S1', 'i1', 'u1', 'i2', 'u2', 'i4', 'u4', 'i8', 'u8', 'f4', 'f8']), got b1

>>> da = xr.DataArray(range(5), attrs={'test': pd.to_datetime('now')})
>>> da
<xarray.DataArray (dim_0: 5)>
array([0, 1, 2, 3, 4])
Dimensions without coordinates: dim_0
Attributes:
    test:     2017-08-03 13:02:29

>>> da.to_netcdf('test_dt.nc')
...
TypeError: Invalid value for attr: 2017-08-03 13:02:29 must be a number string, ndarray or a list/tuple of numbers/strings for serialization to netCDF files

I assume bool attributes aren't supported by netcdf4-python and dates are difficult (could always just write these as a string), but this would be really nice to have if possible.

As an aside, using h5netcdf works for bools, but coerces them to int64.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the DataArray.to_netcdf entry point and reproduce the boolean and pandas datetime examples from the issue using the netcdf4 engine. Trace the netCDF4 serialization path and determine the expected representation for each attribute type; done means both examples serialize successfully without the reported TypeErrors.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.