NVIDIA / NVIDIA/cudf

[PERF/ENH] `Index.intersection` does more hashing work than necessary

Open
#14,487 0 comments 0 reactions 0 assignees View on GitHub
Performance Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Index intersection performs an inner merge of the unique values of the left and right indices (the unique is done so that indices with repeated values don't blow up the memory footprint). This does a full hash of both indices, then the merge (hashing again). Finally, if requested, the result is sorted.

This could be replaced, I think with positive performance effect by either:

- `leftsemi` join + `drop_duplicates`
- `libcudf.search.contains` + `apply_boolean_mask` + `drop_duplicates`

One would have to think through the consequences of either of these wrt any ordering guarantees we might want when `sort=False` (possibly gated behind pandas-compat mode).

This applies _mutatis mutandis_ to `MultiIndex.intersection` too.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.