pydata / pydata/xarray

Calling Dataset.from_dataframe with single-level MultiIndex incorrectly orders data

Open
#3,798 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

This might be a bit of a corner case - or misunderstanding of how to use the relevant methods - but it tripped me up when working with arbitrary-dimension dataframes/datasets and setting up a pandas.MultiIndex.from_product then calling to_xarray on the resultant dataframe.

The case here is one-dimensional data (i.e. a single-level MultiIndex) with out-of-order index/coordinates labels.

MCVE Code Sample
import xarray
import pandas

#
# Create a DataFrame with a single-level MultiIndex, where the labels are not in alphabetical
# order
#
index_multi = pandas.MultiIndex.from_product(
    [['b', 'a', 'c']], 
    names=['test_multi']
)
df_multi = pandas.DataFrame({'test': [1,2,3]}, index=index_multi)

print(df_multi)
#             test
# test_multi
# b              1
# a              2
# c              3

# Convert to Dataset
xr_multi = xarray.Dataset.from_dataframe(df_multi)

#
# The index values have been sorted, but the data values have not been matched
#
print(xr_multi)
# <xarray.Dataset>
# Dimensions:     (test_multi: 3)
# Coordinates:
#   * test_index  (test_multi) object 'a' 'b' 'c'
# Data variables:
#     test        (test_multi) int64 1 2 3

assert xr_multi.test.sel(test_multi='a').data == 2
assert xr_multi.test.sel(test_multi='b').data == 1
assert xr_multi.test.sel(test_multi='c').data == 3
Expected Output

I would expect the assertions to pass - either the coordinates labels not to be sorted, or the data to be reordered to match. Similar examples work fine with a simple Index or two-level MultiIndex:

#
# For reference, the desired behaviour with a simple Index
#
index_simple = pandas.Index(
    ['b', 'a', 'c'], 
    name='test_simple'
)
df_simple = pandas.DataFrame({'test': [1,2,3]}, index=index_simple)

print(df_simple)
#              test
# test_simple
# b               1
# a               2
# c               3

xr_simple = xarray.Dataset.from_dataframe(df_simple)
print(xr_simple)
# <xarray.Dataset>
# Dimensions:      (test_simple: 3)
# Coordinates:
#   * test_simple  (test_simple) object 'b' 'a' 'c'
# Data variables:
#     test         (test_simple) int64 1 2 3

assert xr_simple.test.sel(test_simple='a').data == 2
assert xr_simple.test.sel(test_simple='b').data == 1
assert xr_simple.test.sel(test_simple='c').data == 3


#
# For reference, the desired behavior with a two-level MultiIndex
#
index_multi2 = pandas.MultiIndex.from_tuples(
    [('b', 'b'), ('a', 'a'), ('c', 'c')], 
    names=['test_multi1', 'test_multi2']
)
df_multi2 = pandas.DataFrame({'test': [1,2,3]}, index=index_multi2)

print(df_multi2)
#                          test
# test_multi1 test_multi2
# b           b               1
# a           a               2
# c           c               3

# Convert to Dataset
xr_multi2 = xarray.Dataset.from_dataframe(df_multi2)

#
# The index values have been sorted, and data is reordered and filled out with nans
#
print(xr_multi2)
# <xarray.Dataset>
# Dimensions:      (test_multi1: 3, test_multi2: 3)
# Coordinates:
#   * test_multi1  (test_multi1) object 'a' 'b' 'c'
#   * test_multi2  (test_multi2) object 'a' 'b' 'c'
# Data variables:
#     test         (test_multi1, test_multi2) float64 2.0 nan nan ... nan nan 3.0


assert xr_multi2.test.sel(test_multi1='a', test_multi2='a').data == 2
assert xr_multi2.test.sel(test_multi1='b', test_multi2='b').data == 1
assert xr_multi2.test.sel(test_multi1='c', test_multi2='c').data == 3
Problem Description

Creating a Dataset from a DataFrame with a single-level MultiIndex (where the labels are not in alphabetical order) results in the index/coordinates labels being sorted, but the data values are not reordered to match.

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.8.1 | packaged by conda-forge | (default, Jan 29 2020, 14:24:10) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: English_United Kingdom.1252
libhdf5: None
libnetcdf: None

xarray: 0.15.0
pandas: 1.0.1
numpy: 1.18.1
scipy: 1.4.1
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.2.0rc3
cartopy: None
seaborn: None
numbagg: None
setuptools: 45.2.0.post20200209
pip: 20.0.2
conda: None
pytest: 5.3.5
IPython: None
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the Dataset.from_dataframe entry point and reproduce the MCVE using pandas.MultiIndex.from_product with the labels b, a, and c. Compare the resulting coordinate order and data values with the assertions in the issue; done means the single-level MultiIndex preserves matching labels and values, while the simple Index and two-level MultiIndex behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.