apache / apache/datafusion

Projection rewrites do not rebase physical LambdaVariable bindings across schema boundaries

Open
#24,372 3 comments 0 reactions 1 assignee Claimed by @shinzoxD View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Physical lambda expressions can fail during execution after an optimizer rewrite moves the expression across a schema boundary.

This affects at least two paths:

1. collapsing consecutive `ProjectionExec`s;
2. pushing a join-filter expression below a `NestedLoopJoinExec`.

In both cases, the query plans successfully, but the physical `LambdaVariable` retains an index and `FieldRef` that no longer match the `RecordBatch` against which it is evaluated.

### To Reproduce

#### Reproducer 1: projection-chain collapse

```sql
SET datafusion.sql_parser.dialect = spark;

SELECT array_transform(arr, x -> x)
FROM (
SELECT arr
FROM (VALUES ([1, 2], 7)) AS t(arr, padding)
) AS q;
```

Actual result:

```text
Error: Execution error: Field of physical LambdaVariable with index 0 doesn't match batch field during evaluation Field { "x": nullable Int64 } != Field { "column2": Int64 }
```

The optimized physical plan has collapsed the intermediate projection:

```text
ProjectionExec
array_transform(column1, (x) -> x@1)
DataSourceExec
```

#### Reproducer 2: NestedLoopJoin filter pushdown

```sql
SET datafusion.sql_parser.dialect = spark;

SELECT l.id, r.k
FROM (
VALUES
(1, [1], 10, 20),
(2, arrow_cast([NULL], 'List(Int64)'), 30, 40)
) AS l(id, arr, padding_1, padding_2)
JOIN (VALUES (0), (1), (2)) AS r(k)
ON (cardinality(array_filter(l.arr, x -> x IS NOT NULL)) > 0)
= (l.id > r.k)
ORDER BY l.id, r.k;
```

Actual result:

```text
Error: Execution error: Field of physical LambdaVariable with index 0 doesn't match batch field during evaluation Field { "x": nullable Int64 } != Field { "column4": Int64 }
```

`EXPLAIN` confirms that this goes through `NestedLoopJoinExec` and join-filter projection pushdown:

```text
NestedLoopJoinExec
ProjectionExec
arr: column2
id: column1
join_proj_push_down_1:
cardinality(
array_filter(column2, (x) -> x@3 IS NOT NULL)
) > 0
```

### Expected behavior

The first query should return:

```text
[1, 2]
```

The second query should return:

```text
1, 0
2, 2
```

### Additional context

Related **https://github.com/lakehq/sail/pull/2413**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.