apache / apache/datafusion

Pushdown projection below NestedLoopJoinExec

Open
#14,161 4 comments 0 reactions 1 assignee Claimed by @Rachelint View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

A solid example of what I am looking for is, these plans will project a&c (first CsvExec), and b only (second CsvExec). Embedding the projection into CsvExec is already done, we just need to pushdown the projection below NestedLoopJoinExec

_Originally posted by @berkaysynnada in https://github.com/apache/datafusion/pull/14120#discussion_r1916073985_

Current optimization is like
```rust
let expected = [
"NestedLoopJoinExec: join_type=Inner, filter=a@0 < b@1, projection=[c@2]",
" CsvExec: file_groups={1 group: [[x]]}, projection=[a, b, c, d, e], has_header=false",
" CsvExec: file_groups={1 group: [[x]]}, projection=[a, b, c, d, e], has_header=false",
];
```

where we could further pushdown projection to CscExec
```rust
let expected = [
"NestedLoopJoinExec: join_type=Inner, filter=a@0 < b@1, projection=[c@2]",
" CsvExec: file_groups={1 group: [[x]]}, projection=[a, c], has_header=false",
" CsvExec: file_groups={1 group: [[x]]}, projection=[b], has_header=false",
];
```

Since we only requires `a, c` from left side and `b` from right side

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.