Expression `column_refs` does not return outer referenced columns in subqueries
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Calling `column_refs()` on a subquery expression returns no column references, even if the subquery contains outer references.
PushDownFilter optimization uses the `column_refs()` to compare against the columns returned from `prevent_predicate_push_down_columns()` on an extension node. The effect of this is that a subquery in a filter may get pushed below an extension node when it shouldn't.
### To Reproduce
```rust
let outer_ref = Expr::OuterReferenceColumn(DataType::Int64, Column::new(None::, "a"));
let plan = LogicalPlanBuilder::empty(true)
.project(vec![outer_ref.clone()])
.unwrap()
.build()
.unwrap();
let subq = Subquery {
subquery: Arc::new(plan),
outer_ref_columns: vec![outer_ref],
};
let expr = Expr::Exists(Exists::new(subq, false));
println!("{:?}", expr.column_refs());
```
### Expected behavior
`column_refs()` should return the outer columns used by the subquery expression.
### Additional context
The relevant code is [here](https://github.com/apache/datafusion/blob/main/datafusion/expr/src/expr.rs#L1533). It just collects `Expr::Column`s within self.
Contributor guide
Research direction
Start in datafusion/expr/src/expr.rs around line 1533 and reproduce the issue with the example expression containing an outer reference inside a subquery. Trace how column_refs() collects references, then verify that it returns the subquery's outer columns and that the regression is covered by relevant expression tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100