[FEA] Add optional row-validity masks to hash join and groupby APIs
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
GQE preserves filter results as the original columns plus one row-selection mask. Some GQE join and groupby implementations consume this mask directly, but the corresponding libcudf operators cannot. Using libcudf requires GQE to materialize selected rows or adapt the selection through column null masks.
`apply_boolean_mask` materializes the selected rows in new column buffers. Using it before an operator therefore adds an extra copy of the selected input columns.
Merging the selection into column null masks avoids copying the input columns, but it is not equivalent for every operator.
A row-selection mask controls whether an input row participates in an operator. It is independent of the null validity of values in a participating row.
**Describe the solution you'd like**
Add experimental, optional row-selection mask inputs to hash join and groupby APIs. The initial API may accept either a BOOL8 column or a bitmap -- open to discussing more on the representation to use for the validity mask/ selectivity vector.
The required behavior is operator-specific:
- **Hash join:** Accept masks for the `hash_join` build input, each `streaming_hash_join::insert()` input, and probe inputs. Excluded rows cannot match or appear in the result. Returned row indices continue to refer to the original, unfiltered inputs. Selected key nulls retain the existing `null_equality` behavior.
- **Groupby:** Accept one operator-wide mask for standard groupby and each `streaming_groupby` input batch. Excluded rows neither create groups nor contribute to aggregations. Selected null keys and values retain the existing groupby null semantics.
**Describe alternatives you've considered**
The current alternatives are `apply_boolean_mask` and changing column null masks. Their limitations are described above.
**Additional context**
Maybe rel Issue [#22947](https://github.com/rapidsai/cudf/issues/22947).
Could be extended to partitioning, sort/distinct, and AST/JIT expression evaluation if benchmarks show a benefit.
Contributor guide
Research direction
Read the existing hash_join, streaming_hash_join::insert(), standard groupby, and streaming_groupby APIs, along with related issue #22947, to understand current input and row-index behavior. Define the mask representation and verify the operator-specific rules: excluded rows do not participate, while selected nulls retain existing semantics and returned join indices reference original inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100