apache / apache/datafusion

Optimize `try_embed_to_hash_join` in the ProjectionPushdown rule

Open
#9,483 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

From https://github.com/apache/arrow-datafusion/pull/9236/files#r1513524992,

This is a TODO in datafusion/core/src/physical_optimizer/optimizer.rs:

> TODO: `try_embed_to_hash_join` in the ProjectionPushdown rule would be block by the CoalesceBatches, so add it before CoalesceBatches. Maybe optimize it in the future.

We can optimize `try_embed_to_hash_join` in `ProjectionPushdown` rule to make it unblock by `CoalesceBatches`.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Read datafusion/core/src/physical_optimizer/optimizer.rs and locate try_embed_to_hash_join in the ProjectionPushdown rule. Trace how CoalesceBatches currently blocks it and review the surrounding optimizer flow. Done means the function is optimized so it is no longer blocked by CoalesceBatches, with the relevant optimizer behavior verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.