logical_plan optimize for MemTable will cost more time when collect dataframe
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I try to read table from MemTable which is large, but when I collect it, datafusion cost more time then expectation.
I find this is because logical_plan optimize.
https://github.com/apache/arrow-datafusion/blob/f9f40bf70a9b6af4345f16b906bcf8a839d5f511/datafusion/core/src/execution/context.rs
```rust
// line 1940
pub async fn create_physical_plan(
&self,
logical_plan: &LogicalPlan,
) -> Result> {
let logical_plan = self.optimize(logical_plan)?;
self.query_planner
.create_physical_plan(&logical_plan, self)
.await
}
```
I find old_plan is TableScan{.., projection: None, ..}, but new_plan is TableScan{.., projection: Some([0,1,2,...]), ..}.
when I collect data, datafusion use new_plan will cost more time then use old_plan because of MemoryStream::poll_next:
https://github.com/apache/arrow-datafusion/blob/f9f40bf70a9b6af4345f16b906bcf8a839d5f511/datafusion/core/src/physical_plan/memory.rs
```rust
// line 203
let batch = match self.projection.as_ref() {
Some(columns) => batch.project(columns)?,
None => batch.clone(),
};
```
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start in datafusion/core/src/execution/context.rs at create_physical_plan and compare the old and optimized TableScan plans described in the issue. Then inspect datafusion/core/src/physical_plan/memory.rs at MemoryStream::poll_next and measure collection behavior for a large MemTable with and without projection. Done means the optimized path no longer adds the reported collection cost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100