NVIDIA / NVIDIA/cudf

Remove partial support for duplicate MultIindex names unless they are all None

Open
#10,500 10 comments 1 reaction 0 assignees View on GitHub
breaking improvement Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Currently our MultiIndex class supports duplicate names, while DataFrames do not. The MultiIndex support is buggy, however, and we are frequently finding new edge cases that break it. Since pandas DataFrames do support duplicate names and [we explicitly choose not to](https://github.com/rapidsai/cudf/blob/branch-22.04/python/cudf/cudf/core/dataframe.py#L4377), I think it makes sense to do the same for MultiIndex. It improves our internal consistency and helps us write much more robust code. Making this change would probably fix a number of currently unknown/hidden bugs.

The major caveat here is that we do need to support MultiIndexes where all the names are None. However, handling this case would potentially be much simpler since we could use a sentinel or another class attribute to track whether names are meaningful or not. Default names could be integers, and any setting of names would require setting all column names to unique values.

An aside: If we ever did want to support duplicate names properly, it would involve a refactoring at the level of `ColumnAccessor`, which currently uses a dictionary as the underlying data structure to map names to columns. We would then need to update all of our functions that rely on `_from_data` to populate a new object that could support duplicate names rather than a dictionary. This is a substantial undertaking and out of scope for this issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.