[FEA] Reusable filtered joins in Java for semi/anti joins
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
cudf-spark would like to be able to cache and reuse broadcast hash builds across stream batches. Currently the Java APIs for left semi/anti joins construct a filtered join for each call.
**Describe the solution you'd like**
We can wrap the `libcudf::filtered_join` in Java/JNI, and add `Table.leftSemiJoinGatherMap` and `Table.leftAntiJoinGatherMap` APIs that accept the reusable `FilteredJoin` object. This would closely follow what has been done for HashJoin and DistinctHashJoin.
This request just concerns equi-joins, additional residual predicates would be out of scope.
**Describe alternatives you've considered**
Reusable semi/anti joins can be emulated with a reusable HashJoin to perform an inner join, and turning the result into a semi/anti gather maps, at the cost of enormous intermediate gather maps if there are many duplicates.
**Additional context**
Contributor guide
Research direction
Start by locating the existing Java/JNI bindings for libcudf::filtered_join and the HashJoin and DistinctHashJoin implementations. Trace how Table.leftSemiJoinGatherMap and Table.leftAntiJoinGatherMap currently construct filtered joins, then define the reusable FilteredJoin path. Done means reusable equi-join gather-map APIs are exposed for both semi and anti joins, with residual predicates remaining out of scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100