apache / apache/datafusion

Cost Model

Open
#18,259 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

This task is part of epic #18249

## Cost Model

Do we want to pursue a traditional, complex cost model that estimates all work in a plan, or take a simpler approach? In practice, even the most detailed cost models often prove inaccurate despite significant effort.

Do we want to adopt a more intuitive approach—similar to the join ranking strategy. Consider the following examples:

- Is a plan with two merge joins better than one merge join and one hash join? How should we assign weights and make comparisons?
- Is a merge join on a single stream consistently faster than a partitioned hash join across multiple streams? How do we evaluate and rank these scenarios?
- Instead of using exact byte sizes, could we categorize input sizes as small, medium, or large and assign weights accordingly?

This could serve as a quick and practical research project: define relevant properties, assign weights and criteria, and run simple experiments to compare estimated costs against actual runtimes.

Contributor guide

Open the contributing guide

Research direction

Start with epic #18249 and the Cost Model questions in this issue; no source file, test, or entry point is identified. Define relevant plan properties, weights, and criteria, then run experiments comparing estimated costs with actual runtimes.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.