apache / apache/datafusion

Representing statistics in logical or physical plans (or in both of them)

Open
#4,003 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

There seem to be a few use cases that can be uncovered by being able to have statistics in the logical plan:

- Helping projects (e.g. Dask SQL) that only use logical plans to have a way to access statistics (so that they can actually leverage them themselves, without needing to use the physical plan from DataFusion) (by @andygrove).
- Choosing optimal physical plans at the planning stage (instead of the physical plan optimization stage) (by @mingmwang)
- Easier join analysis when dealing with join trees (in terms of representation) and easier nested join reordering (in terms of the ease of doing complex rewrites).
- A clear separation between logical cost (a cost-based optimization independent from the specifics of any physical operator implementation; e.g. reducing the join sizes with reordering) vs physical cost (something that only relates to implementation of our physical operators; e.g. hash join probe switching)

**Describe the solution you'd like**
There were 3 options proposed in the discussion:
- Represent statistics solely in the logical planning stage (basically a theoretical revert of #962).
- Keep them as is (in the physical plan), and think about a more generic way to convert between logical<->physical plans (probably wouldn't help directly to Dask SQL, but might have other use cases)
- Represent them in both places, with utility toolkits to do generic join/filter selectivity analysis as well as other fundamental cost estimations.

**Describe alternatives you've considered**
Keep them as is, and not do anything else.

**Additional context**
This is a spin-off from the cost calculations/estimations in #3929 (also related to #3983 and #3984). Original discussion [can be found here](https://docs.google.com/document/d/1M4mmV7KA1LSj-D-WJA338B4ydlm-8A8D5OPuDE5_SD4/edit?disco=AAAAiKrO_9c) by @mingmwang @isidentical @Dandandan @jackwener @andygrove @alamb. It also includes a lot of material regarding what other query engine / database systems are doing (so recommend reading it, this is just a main summary to continue to discussions in a more structured/public place).

Contributor guide

Open the contributing guide

Research direction

Start by reading the cost calculations in #3929, the related issues #3983 and #3984, and the linked discussion. Compare the three proposed representations for logical and physical plan statistics, then establish which approach the project wants and what utility analyses are required. Done means the representation and scope are agreed clearly enough to guide implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.