Projection rewrites do not rebase physical LambdaVariable bindings across schema boundaries
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Physical lambda expressions can fail during execution after an optimizer rewrite moves the expression across a schema boundary.
This affects at least two paths:
1. collapsing consecutive `ProjectionExec`s;
2. pushing a join-filter expression below a `NestedLoopJoinExec`.
In both cases, the query plans successfully, but the physical `LambdaVariable` retains an index and `FieldRef` that no longer match the `RecordBatch` against which it is evaluated.
### To Reproduce
#### Reproducer 1: projection-chain collapse
```sql
SET datafusion.sql_parser.dialect = spark;
SELECT array_transform(arr, x -> x)
FROM (
SELECT arr
FROM (VALUES ([1, 2], 7)) AS t(arr, padding)
) AS q;
```
Actual result:
```text
Error: Execution error: Field of physical LambdaVariable with index 0 doesn't match batch field during evaluation Field { "x": nullable Int64 } != Field { "column2": Int64 }
```
The optimized physical plan has collapsed the intermediate projection:
```text
ProjectionExec
array_transform(column1, (x) -> x@1)
DataSourceExec
```
#### Reproducer 2: NestedLoopJoin filter pushdown
```sql
SET datafusion.sql_parser.dialect = spark;
SELECT l.id, r.k
FROM (
VALUES
(1, [1], 10, 20),
(2, arrow_cast([NULL], 'List(Int64)'), 30, 40)
) AS l(id, arr, padding_1, padding_2)
JOIN (VALUES (0), (1), (2)) AS r(k)
ON (cardinality(array_filter(l.arr, x -> x IS NOT NULL)) > 0)
= (l.id > r.k)
ORDER BY l.id, r.k;
```
Actual result:
```text
Error: Execution error: Field of physical LambdaVariable with index 0 doesn't match batch field during evaluation Field { "x": nullable Int64 } != Field { "column4": Int64 }
```
`EXPLAIN` confirms that this goes through `NestedLoopJoinExec` and join-filter projection pushdown:
```text
NestedLoopJoinExec
ProjectionExec
arr: column2
id: column1
join_proj_push_down_1:
cardinality(
array_filter(column2, (x) -> x@3 IS NOT NULL)
) > 0
```
### Expected behavior
The first query should return:
```text
[1, 2]
```
The second query should return:
```text
1, 0
2, 2
```
### Additional context
Related **https://github.com/lakehq/sail/pull/2413**
Contributor guide
Assessment
This issue has not been assessed yet.