Cannot create Dataset from DataFrame containing CategoricalIndex with Intervals
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened:
Creating a Dataset from a DataFrame with a CategoricalIndex (where the categories are an IntervalIndex) raises the error:
TypeError: data type not understood
What you expected to happen:
I expected the output to be a Dataset object with the CategoricalIndex as one of the coordinates.
Minimal Complete Verifiable Example:
import pandas as pd
import xarray as xr
df = pd.DataFrame({"name": list("abcdefghijk"), "age": range(11)})
df["age_group"] = pd.cut(df["age"], bins=5)
df = df.set_index(["name", "age_group"])
df.to_xarray() # or xr.Dataset.from_dataframe(df)
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.7.5 (default, Oct 25 2019, 10:52:18)
[Clang 4.0.1 (tags/RELEASE_401/final)]
python-bits: 64
OS: Darwin
OS-release: 19.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.1
xarray: 0.15.1
pandas: 1.0.3
numpy: 1.18.1
scipy: 1.4.1
netCDF4: 1.4.2
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.1.3
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.1.0
cfgrib: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.1.3
cartopy: None
seaborn: 0.9.0
numbagg: None
setuptools: 46.4.0.post20200518
pip: 20.0.2
conda: 4.8.3
pytest: None
IPython: 7.13.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided pandas DataFrame example through DataFrame.to_xarray() and xr.Dataset.from_dataframe(). Trace the DataFrame conversion entry point and its handling of a CategoricalIndex whose categories are intervals. Done means both conversions produce a Dataset with the CategoricalIndex represented as a coordinate instead of raising TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100