NVIDIA / NVIDIA/cudf

[BUG] `Index.union` does not match pandas for indexes with duplicate entries

Open
#14,488 0 comments 0 reactions 0 assignees View on GitHub
bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

Pandas treats, for `Index.union` only, indexes with duplicate entries as [multisets](https://en.wikipedia.org/wiki/Multiset#Basic_properties_and_operations), for which the union operation produces multiplicities that are the max of the left and right multiplicities. I've asked for clarification of this here https://github.com/pandas-dev/pandas/issues/56137, since it is _only_ `union` that uses the multiset definitions.

cudf, in contrast, performs the union as an outer join. Which produces multiplicities that are the product of the multiplicities of the left and right indexes (with identity for missing values of 1). This matches the pandas behaviour in the case where all entries have multiplicity greater than one in exactly one of the left or right indexes. However, if an entry has a multiplicity larger than one in both left and right indexes, we get the wrong answer.

**Steps/Code to reproduce bug**

```
import cudf
import pandas as pd

left = pd.Index([1, 1])
right = pd.Index([1, 2, 1, 1])
print(left.union(right))

cleft = cudf.from_pandas(left)
cright = cudf.from_pandas(right)

print(cleft.union(cright))
# Int64Index([1, 1, 1, 2], dtype='int64')
# Int64Index([1, 1, 1, 1, 1, 1, 2], dtype='int64')
```

**Expected behavior**

Match pandas.

This can be done with `value_counts`/`merge`/`repeat` in some combo, but there's probably a slightly smarter way.

**Notes**

Also applies to `MultiIndex`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.