apache / apache/datafusion

Expression `column_refs` does not return outer referenced columns in subqueries

Open
#15,046 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Calling `column_refs()` on a subquery expression returns no column references, even if the subquery contains outer references.

PushDownFilter optimization uses the `column_refs()` to compare against the columns returned from `prevent_predicate_push_down_columns()` on an extension node. The effect of this is that a subquery in a filter may get pushed below an extension node when it shouldn't.

### To Reproduce

```rust
let outer_ref = Expr::OuterReferenceColumn(DataType::Int64, Column::new(None::, "a"));
let plan = LogicalPlanBuilder::empty(true)
.project(vec![outer_ref.clone()])
.unwrap()
.build()
.unwrap();

let subq = Subquery {
subquery: Arc::new(plan),
outer_ref_columns: vec![outer_ref],
};

let expr = Expr::Exists(Exists::new(subq, false));

println!("{:?}", expr.column_refs());
```

### Expected behavior

`column_refs()` should return the outer columns used by the subquery expression.

### Additional context

The relevant code is [here](https://github.com/apache/datafusion/blob/main/datafusion/expr/src/expr.rs#L1533). It just collects `Expr::Column`s within self.

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/expr/src/expr.rs around line 1533 and reproduce the issue with the example expression containing an outer reference inside a subquery. Trace how column_refs() collects references, then verify that it returns the subquery's outer columns and that the regression is covered by relevant expression tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.