apache / apache/datafusion-comet

Extended explain operator stats miscount reuse wrappers and CometSubqueryBroadcast

Open
#5,203 0 comments 0 reactions 1 assignee Claimed by @andygrove View on GitHub
bug priority:low
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Describe the bug

The operator coverage summary produced by `ExtendedExplainInfo` (`Comet accelerated N out of M eligible operators (X%)`) counts three node kinds incorrectly, so it understates acceleration on most TPC-DS plans and double-counts reused exchanges.

All of the following are in `ExtendedExplainInfo.generateTreeString`, `spark/src/main/scala/org/apache/comet/ExtendedExplainInfo.scala`.

### 1. `ReusedSubqueryExec` is counted as an un-accelerated Spark operator

The ignore list at line 107 covers `AdaptiveSparkPlanExec | InputAdapter | QueryStageExec | WholeStageCodegenExec | ReusedExchangeExec | AQEShuffleReadExec`. `ReusedSubqueryExec` is not in it, is not a transition type, and is not a `CometPlan`, so it falls through to `case _ => planStats.sparkOperators += 1` (line 116).

`getActualPlan` (line 234) also does not unwrap it, and `ReusedSubqueryExec` is a `LeafExecNode` (`children` is `Nil`), so the subquery plan it points at is never traversed. Net effect per reuse site: one phantom un-accelerated operator, and the native plan behind it contributes nothing.

### 2. `CometSubqueryBroadcastExec` is counted as a Spark operator

`CometSubqueryBroadcastExec` is declared

```scala
case class CometSubqueryBroadcastExec(...)
extends BaseSubqueryExec
with UnaryExecNode {
```

(`spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryBroadcastExec.scala`) and never mixes in `CometPlan`, so despite being Comet's own DPP operator it lands in the Spark bucket at line 116.

### 3. Reused exchanges are counted twice (inconsistent with the above)

`ReusedExchangeExec` is both ignored by the match *and* unwrapped by `getActualPlan` to its child (line 240), so the reused subtree is traversed and counted again in full at every reference.

The two reuse wrappers therefore skew in opposite directions: exchange reuse inflates the operator counts, subquery reuse deflates the acceleration percentage.

## Steps to reproduce

Any plan with DPP or a reused scalar subquery. The checked-in goldens show it directly — `spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q1/extended.txt`:

```
Comet accelerated 47 out of 49 eligible operators (95%). Final plan contains 1 transitions between Spark and Comet.
```

That tree has 50 nodes. One is a transition (`CometColumnarToRow`), leaving 49 eligible, reported as 47 Comet plus 2 Spark. The two "Spark" operators are exactly `ReusedSubquery` and `CometSubqueryBroadcast`; every other node in the plan is a `Comet*` operator mixing in `CometPlan`. The plan is fully accelerated but reports 95%.

Prevalence across the 158 checked-in `extended.txt` goldens:

- `ReusedSubquery` appears in 78
- `CometSubqueryBroadcast` appears in 130
- `ReusedExchange` appears in 0, because `getActualPlan` always unwraps it — which is what makes the double counting invisible in the goldens

## Expected behavior

`ReusedSubqueryExec` is pure bookkeeping, not an operator that does work, so it should be ignored like `ReusedExchangeExec` rather than counted as un-accelerated. `CometSubqueryBroadcastExec` should count as a Comet operator. Reused exchanges should be counted once, not once per reference.

## Additional context

The same counts feed the `comet` metrics source via `CometMetricsListener` -> `CometSource` (`operators.native`, `operators.spark`, `acceleration.ratio`), so the skew is not confined to `EXPLAIN` output — it also biases the acceleration ratio gauge.

Two open design questions for whoever picks this up:

1. For `CometSubqueryBroadcastExec`, mixing in `CometPlan` is the tidier fix but has a wider blast radius, since other rules pattern-match on `CometPlan`. Special-casing it in the explain match is contained but leaves the type lying about itself. Worth checking the other `CometPlan` match sites before choosing.
2. Fixing item 3 means tracking already-visited exchange subtrees during the traversal, which is more involved than items 1 and 2 and could reasonably be split out.

Any of these changes the operator counts, so the TPC-DS plan stability goldens will need regenerating for all supported Spark versions (`./dev/regenerate-golden-files.sh`).

Found while working on #5201, which adds expression-level coverage to the same summary line but does not touch the operator categorization.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.