NVIDIA / NVIDIA/cudf

[FEA] Add optional row-validity masks to hash join and groupby APIs

Open
#23,432 1 comment 0 reactions 0 assignees View on GitHub
feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
GQE preserves filter results as the original columns plus one row-selection mask. Some GQE join and groupby implementations consume this mask directly, but the corresponding libcudf operators cannot. Using libcudf requires GQE to materialize selected rows or adapt the selection through column null masks.

`apply_boolean_mask` materializes the selected rows in new column buffers. Using it before an operator therefore adds an extra copy of the selected input columns.

Merging the selection into column null masks avoids copying the input columns, but it is not equivalent for every operator.

A row-selection mask controls whether an input row participates in an operator. It is independent of the null validity of values in a participating row.

**Describe the solution you'd like**
Add experimental, optional row-selection mask inputs to hash join and groupby APIs. The initial API may accept either a BOOL8 column or a bitmap -- open to discussing more on the representation to use for the validity mask/ selectivity vector.

The required behavior is operator-specific:

- **Hash join:** Accept masks for the `hash_join` build input, each `streaming_hash_join::insert()` input, and probe inputs. Excluded rows cannot match or appear in the result. Returned row indices continue to refer to the original, unfiltered inputs. Selected key nulls retain the existing `null_equality` behavior.
- **Groupby:** Accept one operator-wide mask for standard groupby and each `streaming_groupby` input batch. Excluded rows neither create groups nor contribute to aggregations. Selected null keys and values retain the existing groupby null semantics.

**Describe alternatives you've considered**
The current alternatives are `apply_boolean_mask` and changing column null masks. Their limitations are described above.

**Additional context**
Maybe rel Issue [#22947](https://github.com/rapidsai/cudf/issues/22947).

Could be extended to partitioning, sort/distinct, and AST/JIT expression evaluation if benchmarks show a benefit.

Contributor guide

Open the contributing guide

Research direction

Read the existing hash_join, streaming_hash_join::insert(), standard groupby, and streaming_groupby APIs, along with related issue #22947, to understand current input and row-index behavior. Define the mask representation and verify the operator-specific rules: excluded rows do not participate, while selected nulls retain existing semantics and returned join indices reference original inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.