pydata / pydata/xarray

Cannot store data after `groupby_bins`

Open
#2,847 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

topic-groupby
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Hi Xarray!

I really like your Library. But now I am stuck completely.

Code Sample, a copy-pastable example if possible
import numpy as np
import xarray as xr

data = [1,2,3,4,5,6,7,8,9,10]
bins = np.array(range(5)) * 2
xr_data = xr.Dataset({'data': data})
out = xr_data.groupby_bins('data', bins).mean()
out.to_netcdf('/tmp/test')
Problem description

Get Error :

Traceback (most recent call last):
  File "/home/volker/workspace/pycharm-community-2018.1.2/helpers/pydev/pydevd.py", line 1664, in <module>
    main()
  File "/home/volker/workspace/pycharm-community-2018.1.2/helpers/pydev/pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/volker/workspace/pycharm-community-2018.1.2/helpers/pydev/pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/volker/workspace/pycharm-community-2018.1.2/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/volker/workspace/eprofile_wind/eprofile/src/eprofile/sandbox/test_xarray.py", line 12, in <module>
    out.to_netcdf('/tmp/test')
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/core/dataset.py", line 1232, in to_netcdf
    compute=compute)
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/backends/api.py", line 747, in to_netcdf
    unlimited_dims=unlimited_dims)
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/backends/api.py", line 790, in dump_to_store
    unlimited_dims=unlimited_dims)
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/backends/common.py", line 261, in store
    variables, attributes = self.encode(variables, attributes)
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/backends/common.py", line 347, in encode
    variables, attributes = cf_encoder(variables, attributes)
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/conventions.py", line 605, in cf_encoder
    for k, v in iteritems(variables))
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/conventions.py", line 605, in <genexpr>
    for k, v in iteritems(variables))
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/conventions.py", line 241, in encode_cf_variable
    var = ensure_dtype_not_object(var, name=name)
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/conventions.py", line 201, in ensure_dtype_not_object
    data = _copy_with_dtype(data, dtype=_infer_dtype(data, name))
  File "/home/volker/workspace/eprofile_wind-CRxNsezQ/lib/python3.5/site-packages/xarray/conventions.py", line 139, in _infer_dtype
    .format(name))
ValueError: unable to infer dtype on variable 'data_bins'; xarray cannot serialize arbitrary Python objects
Expected Output

The Dataset should be written to file in netcdf

Output of xr.show_versions()
>>> xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516]
python-bits: 64
OS: Linux
OS-release: 4.9.0-8-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8
LOCALE: de_DE.UTF-8
libhdf5: 1.10.2
libnetcdf: 4.4.1.1

xarray: 0.11.3
pandas: 0.24.1
numpy: 1.16.1
scipy: None
netCDF4: 1.4.2
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.0.2.1
PseudonetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
cyordereddict: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the groupby_bins('data', bins).mean() result and follow out.to_netcdf() into the serialization path shown in the traceback, especially the data_bins variable. Confirm the failure with the provided example and make the resulting Dataset writable as NetCDF without arbitrary Python objects.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
backend, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.