apache / apache/datafusion

Expr::apply_children/map_children & LogicalPlan::map_expressions does not correctly map subquery `outer_ref_column` expressions

Open
#16,147 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

The TreeNode implementation for `Expr` does not handle the `outer_ref_column` expressions present in the `Subquery` nodes of `Expr::InSubquery`, `Expr::ScalarSubquery`, and `Expr::Exists`.

The LogicalPlan::map_expressions implementation does not handle the `outer_ref_column` expression in `LogicalPlan::Subquery`

### To Reproduce

Create a LogicalPlan from a SQL statement with a subquery expression that has an outer column reference, i.e.

```
SELECT
e.employee_id,
e.first_name,
e.salary
FROM
employees e
WHERE
e.salary > (
SELECT AVG(salary)
FROM employees e2
WHERE e2.department_id = e.department_id
);
```

I would expect to be able to do something like this:

```rust
let plan = sql_to_plan(sql);
plan.apply(|plan| {
plan.apply_expressions(|expr| {
match expr {
Expr::OuterReferenceColumn(_, _) => { do something useful }
_ => Ok(Transformed::no(expr))
}
});
});
```

### Expected behavior

The `outer_ref_column` expressions are part of the TreeNode visitor implementations in both Expr and LogicalPlan

### Additional context

I will not work on this, so someone should feel free to take this. There is a workaround if you know about this behavior, which is to match on the expressions/plans which contain these outer ref column expressions.

Contributor guide

Open the contributing guide

Research direction

Start with the TreeNode visitor implementations for Expr and LogicalPlan::map_expressions, then reproduce the behavior using the correlated SQL subquery in the issue. Check whether outer_ref_column expressions in InSubquery, ScalarSubquery, Exists, and LogicalPlan::Subquery are visited. Done means the outer reference is exposed to the expression visitor without relying on the documented workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.