NVIDIA / NVIDIA/cudf

[PERF] Improve "isin" performance by only sorting once

Open
#11,548 2 comments 0 reactions 0 assignees View on GitHub
Performance Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
While benchmarking cuDF-python, I noticed that [bench_isin](https://github.com/rapidsai/cudf/blob/65a782112f4b76941483adf17f9a30a6824f6164/python/cudf/benchmarks/API/bench_dataframe.py#L50) has low end-to-end data throughput (<10GB/s). A closer look at the profiles showed that the data is being sorted twice, first with `.sort_values()` and then as part of `drop_duplicates()`. The following profile is for a test dataframe with 1 col and 100K rows, and is uses the `isin` argument `range(1000)` in the `bench_isin` benchmark.

image

When calling `isin` with a dataframe or dict argument, the profile shows two calls to `Frame.argsort`.

image

**Describe the solution you'd like**
For a performance improvement, I'd like to refactor `isin` to only sort the data once. We should prefer the libcudf `unique` function to `drop_duplicates` for pre-sorted data.

**Describe alternatives you've considered**
n/a

**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.