Join operators don't yield to tokio runtime, preventing query cancellation
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Describe the bug
Join operators (HashJoinExec, SortMergeJoinExec, CrossJoinExec, NestedLoopJoinExec) don't yield to the tokio runtime during execution, which prevents query cancellation from working. When a long-running join query is aborted, the cancellation is never processed because the operator never yields control back to tokio.
## To Reproduce
Reproducer project: https://github.com/erratic-pattern/datafusion-join-cancellation-repro
## Expected behavior
Join operators should periodically yield to the tokio runtime so that task cancellation requests are processed promptly and to avoid worker starvation for other tasks.
## Additional context
A simple fix would be to wrap join streams with [`make_cooperative()`](https://github.com/apache/arrow-datafusion/blob/775277ae3cef7a97f0ca4868b4ccdfc03e9507ee/datafusion/physical-plan/src/coop.rs#L335) in each operator's `execute()` method.
**PR**: #19360
Contributor guide
Assessment
This issue has not been assessed yet.