pydata / pydata/xarray

Cannot create Dataset from DataFrame containing CategoricalIndex with Intervals

Open
#4,307 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:
Creating a Dataset from a DataFrame with a CategoricalIndex (where the categories are an IntervalIndex) raises the error:

TypeError: data type not understood

What you expected to happen:
I expected the output to be a Dataset object with the CategoricalIndex as one of the coordinates.

Minimal Complete Verifiable Example:

import pandas as pd
import xarray as xr

df = pd.DataFrame({"name": list("abcdefghijk"), "age": range(11)})
df["age_group"] = pd.cut(df["age"], bins=5)
df = df.set_index(["name", "age_group"])

df.to_xarray()  # or xr.Dataset.from_dataframe(df)

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.5 (default, Oct 25 2019, 10:52:18)
[Clang 4.0.1 (tags/RELEASE_401/final)]
python-bits: 64
OS: Darwin
OS-release: 19.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.1

xarray: 0.15.1
pandas: 1.0.3
numpy: 1.18.1
scipy: 1.4.1
netCDF4: 1.4.2
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.1.3
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.1.0
cfgrib: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.1.3
cartopy: None
seaborn: 0.9.0
numbagg: None
setuptools: 46.4.0.post20200518
pip: 20.0.2
conda: 4.8.3
pytest: None
IPython: 7.13.0
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided pandas DataFrame example through DataFrame.to_xarray() and xr.Dataset.from_dataframe(). Trace the DataFrame conversion entry point and its handling of a CategoricalIndex whose categories are intervals. Done means both conversions produce a Dataset with the CategoricalIndex represented as a coordinate instead of raising TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.