NVIDIA / NVIDIA/cudf

[FEA] Reusable filtered joins in Java for semi/anti joins

Open
#24,144 0 comments 0 reactions 0 assignees View on GitHub
feature request improvement Java
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

cudf-spark would like to be able to cache and reuse broadcast hash builds across stream batches. Currently the Java APIs for left semi/anti joins construct a filtered join for each call.

**Describe the solution you'd like**

We can wrap the `libcudf::filtered_join` in Java/JNI, and add `Table.leftSemiJoinGatherMap` and `Table.leftAntiJoinGatherMap` APIs that accept the reusable `FilteredJoin` object. This would closely follow what has been done for HashJoin and DistinctHashJoin.

This request just concerns equi-joins, additional residual predicates would be out of scope.

**Describe alternatives you've considered**

Reusable semi/anti joins can be emulated with a reusable HashJoin to perform an inner join, and turning the result into a semi/anti gather maps, at the cost of enormous intermediate gather maps if there are many duplicates.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing Java/JNI bindings for libcudf::filtered_join and the HashJoin and DistinctHashJoin implementations. Trace how Table.leftSemiJoinGatherMap and Table.leftAntiJoinGatherMap currently construct filtered joins, then define the reusable FilteredJoin path. Done means reusable equi-join gather-map APIs are exposed for both semi and anti joins, with residual predicates remaining out of scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.