apache / apache/datafusion

Support join filter in NestedLoopJoin in fizz join test cases

Open
#10,787 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

https://github.com/apache/datafusion/pull/10728 closes the first part of #10659, however combining the `on_filter` with the provided `join_filter` produces wrong results,

```
---- fuzz_cases::join_fuzz::test_inner_join_1k_filtered stdout ----
thread 'fuzz_cases::join_fuzz::test_inner_join_1k_filtered' panicked at datafusion\core\tests\fuzz_cases\join_fuzz.rs:387:17:
assertion `left == right` failed: NestedLoopJoinExec and HashJoinExec produced different results
left: (3, "| 21 | 21 | 2078626761 | 1773284061 | 21 | 25 | 1533589294 | 685853846 |")
right: (3, "| 0 | 75 | -1848255860 | 30924674 | 0 | 75 | 1261908710 | 1064557805 |")
stack backtrace:
```

### Describe the solution you'd like

Add the right logic to the nested inner join loop filter

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/core/tests/fuzz_cases/join_fuzz.rs around line 387 and run fuzz_cases::join_fuzz::test_inner_join_1k_filtered. Trace how NestedLoopJoinExec applies the on_filter and provided join_filter, then compare its result with HashJoinExec. Done means the failing inner join case produces matching results without the assertion panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.