[BUG] Sliced DataFrame with MultiIndex removes unused level values
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Sliced DataFrame with MultiIndex removes unused level values unlike pandas
**Steps/Code to reproduce bug**
```
In [1]: import cudf
In [2]: arrays = [
...:
...: ["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],
...:
...: ["one", "two", "one", "two", "one", "two", "one", "two"],
...:
...: ]
...:
...:
...:
...: tuples = list(zip(*arrays))
In [4]: index = cudf.MultiIndex.from_tuples(tuples, names=["first", "second"])
In [6]: import numpy as np
In [7]: df = cudf.DataFrame(np.random.randn(3, 8), index=["A", "B", "C"], columns=index)
In [8]: df
Out[8]:
first bar baz foo qux
second one two one two one two one two
A 0.248351 -0.162527 2.140643 1.358278 -0.435850 -1.279155 0.932680 2.611601
B -1.321047 -0.403212 0.018679 0.404523 0.015960 1.242899 0.642264 -2.499115
C -0.930653 1.753720 0.247863 0.580408 0.227763 -0.565367 -0.868276 1.056060
In [9]: df[["foo","qux"]].columns.levels
Out[9]: FrozenList([['foo', 'qux'], ['one', 'two']])
In [10]: df.to_pandas()[["foo","qux"]].columns.levels
Out[10]: FrozenList([['bar', 'baz', 'foo', 'qux'], ['one', 'two']])
```
**Expected behavior**
I think in pandas this is an optimization similar to Categorical to keep all level values just in case of recomputations (as noted by a `remove_unused_levels` API for `MultiIndex`). cudf might not need this but it doesn't seem to match pandas behavior
**Environment overview (please complete the following information)**
- Environment location: Bare-metal
- Method of cuDF install: conda
- If method of install is [Docker], provide `docker pull` & `docker run` commands used
**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details
**Additional context**
Add any other context about the problem here.
Contributor guide
Assessment
This issue has not been assessed yet.