apache / apache/datafusion-comet
[EPIC] Improve Comet planning
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Comet currently eagerly tries to convert each operator to a Comet operator. This sometimes leads to inefficient plans that can be slower than just falling back to Spark.
One example is where a Comet JVM shuffle is inserted for a child plan that runs in Spark, to convert to columner shuffle, and then the next stage converts back to row-based right away.
```
+- HashAggregate
+- CometNativeColumnarToRow
+- CometColumnarExchange
+- HashAggregate
```
Another example we saw recently was with DPP fallback, and there was a fix applied for that specific case.
I think it is time to start looking at the overall planning strategy.
Document to discuss:
https://docs.google.com/document/d/1ux_SwZPd64VTtuTC9lC46oIZGKcoH6KhWOS8xlqwdx8/edit?usp=sharing
Tracking for related issues:
- https://github.com/apache/datafusion-comet/issues/4004
### Describe the potential solution
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the linked planning document and inspect issue #4004 for related context. The issue names no source files or tests, so first identify the planning entry points and existing fallback cases before defining a measurable planning strategy that avoids unnecessary Spark-to-Comet conversions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100