NVIDIA / NVIDIA/cudf

[BUG] Sliced DataFrame with MultiIndex removes unused level values

Open
#13,865 0 comments 0 reactions 0 assignees View on GitHub
bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Sliced DataFrame with MultiIndex removes unused level values unlike pandas

**Steps/Code to reproduce bug**
```
In [1]: import cudf

In [2]: arrays = [
...:
...: ["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],
...:
...: ["one", "two", "one", "two", "one", "two", "one", "two"],
...:
...: ]
...:
...:
...:
...: tuples = list(zip(*arrays))

In [4]: index = cudf.MultiIndex.from_tuples(tuples, names=["first", "second"])

In [6]: import numpy as np

In [7]: df = cudf.DataFrame(np.random.randn(3, 8), index=["A", "B", "C"], columns=index)

In [8]: df
Out[8]:
first bar baz foo qux
second one two one two one two one two
A 0.248351 -0.162527 2.140643 1.358278 -0.435850 -1.279155 0.932680 2.611601
B -1.321047 -0.403212 0.018679 0.404523 0.015960 1.242899 0.642264 -2.499115
C -0.930653 1.753720 0.247863 0.580408 0.227763 -0.565367 -0.868276 1.056060

In [9]: df[["foo","qux"]].columns.levels
Out[9]: FrozenList([['foo', 'qux'], ['one', 'two']])

In [10]: df.to_pandas()[["foo","qux"]].columns.levels
Out[10]: FrozenList([['bar', 'baz', 'foo', 'qux'], ['one', 'two']])
```

**Expected behavior**
I think in pandas this is an optimization similar to Categorical to keep all level values just in case of recomputations (as noted by a `remove_unused_levels` API for `MultiIndex`). cudf might not need this but it doesn't seem to match pandas behavior

**Environment overview (please complete the following information)**
- Environment location: Bare-metal
- Method of cuDF install: conda
- If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.