Pushdown projection below NestedLoopJoinExec
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
A solid example of what I am looking for is, these plans will project a&c (first CsvExec), and b only (second CsvExec). Embedding the projection into CsvExec is already done, we just need to pushdown the projection below NestedLoopJoinExec
_Originally posted by @berkaysynnada in https://github.com/apache/datafusion/pull/14120#discussion_r1916073985_
Current optimization is like
```rust
let expected = [
"NestedLoopJoinExec: join_type=Inner, filter=a@0 < b@1, projection=[c@2]",
" CsvExec: file_groups={1 group: [[x]]}, projection=[a, b, c, d, e], has_header=false",
" CsvExec: file_groups={1 group: [[x]]}, projection=[a, b, c, d, e], has_header=false",
];
```
where we could further pushdown projection to CscExec
```rust
let expected = [
"NestedLoopJoinExec: join_type=Inner, filter=a@0 < b@1, projection=[c@2]",
" CsvExec: file_groups={1 group: [[x]]}, projection=[a, c], has_header=false",
" CsvExec: file_groups={1 group: [[x]]}, projection=[b], has_header=false",
];
```
Since we only requires `a, c` from left side and `b` from right side
Contributor guide
Assessment
This issue has not been assessed yet.