[PERF] Improve "isin" performance by only sorting once
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
While benchmarking cuDF-python, I noticed that [bench_isin](https://github.com/rapidsai/cudf/blob/65a782112f4b76941483adf17f9a30a6824f6164/python/cudf/benchmarks/API/bench_dataframe.py#L50) has low end-to-end data throughput (<10GB/s). A closer look at the profiles showed that the data is being sorted twice, first with `.sort_values()` and then as part of `drop_duplicates()`. The following profile is for a test dataframe with 1 col and 100K rows, and is uses the `isin` argument `range(1000)` in the `bench_isin` benchmark.

When calling `isin` with a dataframe or dict argument, the profile shows two calls to `Frame.argsort`.

**Describe the solution you'd like**
For a performance improvement, I'd like to refactor `isin` to only sort the data once. We should prefer the libcudf `unique` function to `drop_duplicates` for pre-sorted data.
**Describe alternatives you've considered**
n/a
**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.