apache / apache/datafusion

Join operators don't yield to tokio runtime, preventing query cancellation

Open
#19,358 1 comment 0 reactions 1 assignee Claimed by @erratic-pattern View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Describe the bug

Join operators (HashJoinExec, SortMergeJoinExec, CrossJoinExec, NestedLoopJoinExec) don't yield to the tokio runtime during execution, which prevents query cancellation from working. When a long-running join query is aborted, the cancellation is never processed because the operator never yields control back to tokio.

## To Reproduce

Reproducer project: https://github.com/erratic-pattern/datafusion-join-cancellation-repro

## Expected behavior

Join operators should periodically yield to the tokio runtime so that task cancellation requests are processed promptly and to avoid worker starvation for other tasks.

## Additional context

A simple fix would be to wrap join streams with [`make_cooperative()`](https://github.com/apache/arrow-datafusion/blob/775277ae3cef7a97f0ca4868b4ccdfc03e9507ee/datafusion/physical-plan/src/coop.rs#L335) in each operator's `execute()` method.

**PR**: #19360

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.