Add fuzz tests for SortMergeJoin spilling
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Specifically, for testing, given the subtlety of the code involved I am not 100% sure it works for all corner cases. I suggest (as a follow on) we invest in fuzz testing both for SMJ in general as well as for spilling SMJ
https://github.com/apache/datafusion/blob/6c0e4fb5d9ac7a0a2f2b91f8b88d21f0bc0b4424/datafusion/core/tests/fuzz_cases/join_fuzz.rs#L50-L49
I think in particular, making sure we adjust the random inputs to have different numbers of repeated values (as the code in this PR is only going to be exercised when there are many of the same join keys I think)
_Originally posted by @alamb in https://github.com/apache/datafusion/pull/11218#pullrequestreview-2186617624_
Contributor guide
Assessment
This issue has not been assessed yet.