apache / apache/datafusion

logical_plan optimize for MemTable will cost more time when collect dataframe

Open
#6,006 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I try to read table from MemTable which is large, but when I collect it, datafusion cost more time then expectation.
I find this is because logical_plan optimize.
https://github.com/apache/arrow-datafusion/blob/f9f40bf70a9b6af4345f16b906bcf8a839d5f511/datafusion/core/src/execution/context.rs
```rust
// line 1940
pub async fn create_physical_plan(
&self,
logical_plan: &LogicalPlan,
) -> Result> {
let logical_plan = self.optimize(logical_plan)?;
self.query_planner
.create_physical_plan(&logical_plan, self)
.await
}
```
I find old_plan is TableScan{.., projection: None, ..}, but new_plan is TableScan{.., projection: Some([0,1,2,...]), ..}.
when I collect data, datafusion use new_plan will cost more time then use old_plan because of MemoryStream::poll_next:
https://github.com/apache/arrow-datafusion/blob/f9f40bf70a9b6af4345f16b906bcf8a839d5f511/datafusion/core/src/physical_plan/memory.rs
```rust
// line 203
let batch = match self.projection.as_ref() {
Some(columns) => batch.project(columns)?,
None => batch.clone(),
};
```

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/core/src/execution/context.rs at create_physical_plan and compare the old and optimized TableScan plans described in the issue. Then inspect datafusion/core/src/physical_plan/memory.rs at MemoryStream::poll_next and measure collection behavior for a large MemTable with and without projection. Done means the optimized path no longer adds the reported collection cost.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.