pydata / pydata/xarray

A suggested solution to the `TypeError: Invalid value for attr:` error upon `.to_netcdf`

Open
#3,743 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Even though netcdf conventions don't allow some data types in the attributes, it might be usefull to simply serialize those as strings rather than throw an error. Maybe add a force_serialization keyword argument to the .to_netcdf method.

Example:
Setup a DataArray with bad values in the attributes:

import numpy as np
from xarray import DataArray, load_dataset
from pandas import Timestamp

from numbers import Number

valid_types = (str, Number, np.ndarray, np.number, list, tuple)

da = DataArray(
    name='bad_values',
    attrs=dict(
        bool_value=True,
        none_value=None,
        datetime_value=Timestamp.now()
    )
)

ds = da.to_dataset()
ds.bad_values.attrs

Output:

{'bool_value': True,
 'none_value': None,
 'datetime_value': Timestamp('2020-02-03 10:53:02.350105')}

The code in the except clause can be easily impolemented under _validate_attrs.

try:
    ds.to_netcdf('test.nc')
    # Fails with TypeError: Invalid value for attr: ...
except TypeError as e:
    print(e.__class__.__name__, e)
    for variable in ds.variables.values():
        for k, v in variable.attrs.items():
            if not isinstance(v, valid_types) or isinstance(v, bool):
                variable.attrs[k] = str(v)

    ds.to_netcdf('test.nc')  # Works as expected

ds_from_file = load_dataset('test.nc')
ds_from_file.bad_values.attrs

Output:

TypeError Invalid value for attr: None must be a number, a string, an ndarray or a list/tuple of numbers/strings for serialization to netCDF files

{'bool_value': 'True',
 'none_value': 'None',
 'datetime_value': '2020-02-03 10:43:38.479866'}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided DataArray and to_netcdf example to reproduce the invalid-attribute error. Then read _validate_attrs in xarray/backends/api.py and determine the intended force_serialization behavior, including the round-trip result shown in the issue; the issue does not name a test file.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.