apache / apache/datafusion-comet

Add mode to run Comet planning but execute with Spark, so users can assess potential Comet coverage

Open
#5,335 1 comment 0 reactions 1 assignee Claimed by @coderfender View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## What is the problem the feature request solves?

Users evaluating Comet have no low-risk way to find out how much of their workload Comet would actually accelerate. Enabling Comet changes execution, so any assessment carries correctness and performance risk and usually needs a separate, dedicated run.

## Describe the potential solution

Add a "plan only" / dry-run mode (e.g. `spark.comet.planOnly.enabled`) where Comet runs its full planning pass — operator/expression conversion and fallback analysis — logs the resulting Comet plan, and then executes the original Spark plan instead.

The mechanism can reuse what `spark.comet.exec.transitionRevert.enabled` already does in `RevertNativeForTransitionHeavyStages`: after planning, replace Comet operators with their Spark equivalents. Here it would be applied unconditionally to the whole plan rather than per stage above a transition threshold.

This composes with existing reporting (extended explain fallback/coverage info and `spark.comet.metrics.enabled`), so a user could run their production workload unchanged and collect an estimate of Comet coverage.

Points to discuss:

- Where to surface the Comet plan: driver log, extended explain info, and/or metrics.
- Whether to also serialize/create the native plan, which catches native-side planning failures and gives a more accurate estimate at some planning cost.
- Behavior under AQE, where planning happens per stage.
- Whether this belongs as its own config or as a mode of the existing explain/telemetry features.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.