[BUG] `Index.union` does not match pandas for indexes with duplicate entries
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Pandas treats, for `Index.union` only, indexes with duplicate entries as [multisets](https://en.wikipedia.org/wiki/Multiset#Basic_properties_and_operations), for which the union operation produces multiplicities that are the max of the left and right multiplicities. I've asked for clarification of this here https://github.com/pandas-dev/pandas/issues/56137, since it is _only_ `union` that uses the multiset definitions.
cudf, in contrast, performs the union as an outer join. Which produces multiplicities that are the product of the multiplicities of the left and right indexes (with identity for missing values of 1). This matches the pandas behaviour in the case where all entries have multiplicity greater than one in exactly one of the left or right indexes. However, if an entry has a multiplicity larger than one in both left and right indexes, we get the wrong answer.
**Steps/Code to reproduce bug**
```
import cudf
import pandas as pd
left = pd.Index([1, 1])
right = pd.Index([1, 2, 1, 1])
print(left.union(right))
cleft = cudf.from_pandas(left)
cright = cudf.from_pandas(right)
print(cleft.union(cright))
# Int64Index([1, 1, 1, 2], dtype='int64')
# Int64Index([1, 1, 1, 1, 1, 1, 2], dtype='int64')
```
**Expected behavior**
Match pandas.
This can be done with `value_counts`/`merge`/`repeat` in some combo, but there's probably a slightly smarter way.
**Notes**
Also applies to `MultiIndex`.
Contributor guide
Assessment
This issue has not been assessed yet.