apache / apache/datafusion

Uncancellable FilterExec when the predicate rejects all rows

Open
#21,893 4 comments 0 reactions 1 assignee Claimed by @mpurins-coralogix View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`FilterExecStream::poll_next` re-polls its input via `continue` without yielding when the predicate rejects every row in a batch. With many such batches the task becomes uncancellable and pins a Tokio worker. Worst with expensive predicates or non-cooperative inputs. `EnsureCooperative` does not help because it skips lazy non-leaf operators.

### To Reproduce

I don't have exact reproducer, but we are hitting this for queries which are something like
```
SELECT * FROM unbounded_source
WHERE expensive_udf(col) = 'never_matches';
```

### Expected behavior

`FilterExec` yields cooperatively so that cancellation takes effect within bounded time regardless of predicate cost or whether anything upstream is cooperative.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.