apache / apache/datafusion-comet
Add mode to run Comet planning but execute with Spark, so users can assess potential Comet coverage
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## What is the problem the feature request solves?
Users evaluating Comet have no low-risk way to find out how much of their workload Comet would actually accelerate. Enabling Comet changes execution, so any assessment carries correctness and performance risk and usually needs a separate, dedicated run.
## Describe the potential solution
Add a "plan only" / dry-run mode (e.g. `spark.comet.planOnly.enabled`) where Comet runs its full planning pass — operator/expression conversion and fallback analysis — logs the resulting Comet plan, and then executes the original Spark plan instead.
The mechanism can reuse what `spark.comet.exec.transitionRevert.enabled` already does in `RevertNativeForTransitionHeavyStages`: after planning, replace Comet operators with their Spark equivalents. Here it would be applied unconditionally to the whole plan rather than per stage above a transition threshold.
This composes with existing reporting (extended explain fallback/coverage info and `spark.comet.metrics.enabled`), so a user could run their production workload unchanged and collect an estimate of Comet coverage.
Points to discuss:
- Where to surface the Comet plan: driver log, extended explain info, and/or metrics.
- Whether to also serialize/create the native plan, which catches native-side planning failures and gives a more accurate estimate at some planning cost.
- Behavior under AQE, where planning happens per stage.
- Whether this belongs as its own config or as a mode of the existing explain/telemetry features.
Contributor guide
Assessment
This issue has not been assessed yet.