apache / apache/datafusion

Plan Pruning Strategies

Open
#18,258 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

This task is part of feature #18249

Plan pruning can be adaptable, with several strategies depending on system constraints and goals. Here are a few examples:

- **No pruning until the end:** If enumeration produces a manageable number of partial plans and memory is abundant, we can defer pruning entirely. A cost model is then applied at the final stage to select the best plans.
- **Incremental pruning by level:** To prevent plan space explosion, we retain a subset of plans at each level, using a cost model to discard less promising ones.
- **Memory-aware pruning:** Pruning is triggered only when memory usage exceeds a predefined threshold.
- **Plan count-based pruning:** We prune only when the number of partial plans surpasses a specified limit.
- **Hybrid pruning strategy:** only prune when certain conditions hit and prune to keep certain plans

This flexibility allows pruning strategies to be tailored to system capacity and optimization goals.

Contributor guide

Open the contributing guide

Research direction

Start with feature #18249, which this task belongs to, and review its plan-enumeration requirements before deciding how pruning strategies should be represented. The issue names no files, entry points, tests, limits, or acceptance criteria, so completion cannot be verified from this issue alone.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.