apache / apache/datafusion-comet

[EPIC] Improve Comet planning

Open
#4,005 1 comment 0 reactions 0 assignees View on GitHub
enhancement priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Comet currently eagerly tries to convert each operator to a Comet operator. This sometimes leads to inefficient plans that can be slower than just falling back to Spark.

One example is where a Comet JVM shuffle is inserted for a child plan that runs in Spark, to convert to columner shuffle, and then the next stage converts back to row-based right away.

```
+- HashAggregate
+- CometNativeColumnarToRow
+- CometColumnarExchange
+- HashAggregate
```

Another example we saw recently was with DPP fallback, and there was a fix applied for that specific case.

I think it is time to start looking at the overall planning strategy.

Document to discuss:

https://docs.google.com/document/d/1ux_SwZPd64VTtuTC9lC46oIZGKcoH6KhWOS8xlqwdx8/edit?usp=sharing

Tracking for related issues:

- https://github.com/apache/datafusion-comet/issues/4004

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the linked planning document and inspect issue #4004 for related context. The issue names no source files or tests, so first identify the planning entry points and existing fallback cases before defining a measurable planning strategy that avoids unnecessary Spark-to-Comet conversions.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.