apache / apache/datafusion-comet
Extended explain operator stats miscount reuse wrappers and CometSubqueryBroadcast
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Describe the bug
The operator coverage summary produced by `ExtendedExplainInfo` (`Comet accelerated N out of M eligible operators (X%)`) counts three node kinds incorrectly, so it understates acceleration on most TPC-DS plans and double-counts reused exchanges.
All of the following are in `ExtendedExplainInfo.generateTreeString`, `spark/src/main/scala/org/apache/comet/ExtendedExplainInfo.scala`.
### 1. `ReusedSubqueryExec` is counted as an un-accelerated Spark operator
The ignore list at line 107 covers `AdaptiveSparkPlanExec | InputAdapter | QueryStageExec | WholeStageCodegenExec | ReusedExchangeExec | AQEShuffleReadExec`. `ReusedSubqueryExec` is not in it, is not a transition type, and is not a `CometPlan`, so it falls through to `case _ => planStats.sparkOperators += 1` (line 116).
`getActualPlan` (line 234) also does not unwrap it, and `ReusedSubqueryExec` is a `LeafExecNode` (`children` is `Nil`), so the subquery plan it points at is never traversed. Net effect per reuse site: one phantom un-accelerated operator, and the native plan behind it contributes nothing.
### 2. `CometSubqueryBroadcastExec` is counted as a Spark operator
`CometSubqueryBroadcastExec` is declared
```scala
case class CometSubqueryBroadcastExec(...)
extends BaseSubqueryExec
with UnaryExecNode {
```
(`spark/src/main/scala/org/apache/spark/sql/comet/CometSubqueryBroadcastExec.scala`) and never mixes in `CometPlan`, so despite being Comet's own DPP operator it lands in the Spark bucket at line 116.
### 3. Reused exchanges are counted twice (inconsistent with the above)
`ReusedExchangeExec` is both ignored by the match *and* unwrapped by `getActualPlan` to its child (line 240), so the reused subtree is traversed and counted again in full at every reference.
The two reuse wrappers therefore skew in opposite directions: exchange reuse inflates the operator counts, subquery reuse deflates the acceleration percentage.
## Steps to reproduce
Any plan with DPP or a reused scalar subquery. The checked-in goldens show it directly — `spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q1/extended.txt`:
```
Comet accelerated 47 out of 49 eligible operators (95%). Final plan contains 1 transitions between Spark and Comet.
```
That tree has 50 nodes. One is a transition (`CometColumnarToRow`), leaving 49 eligible, reported as 47 Comet plus 2 Spark. The two "Spark" operators are exactly `ReusedSubquery` and `CometSubqueryBroadcast`; every other node in the plan is a `Comet*` operator mixing in `CometPlan`. The plan is fully accelerated but reports 95%.
Prevalence across the 158 checked-in `extended.txt` goldens:
- `ReusedSubquery` appears in 78
- `CometSubqueryBroadcast` appears in 130
- `ReusedExchange` appears in 0, because `getActualPlan` always unwraps it — which is what makes the double counting invisible in the goldens
## Expected behavior
`ReusedSubqueryExec` is pure bookkeeping, not an operator that does work, so it should be ignored like `ReusedExchangeExec` rather than counted as un-accelerated. `CometSubqueryBroadcastExec` should count as a Comet operator. Reused exchanges should be counted once, not once per reference.
## Additional context
The same counts feed the `comet` metrics source via `CometMetricsListener` -> `CometSource` (`operators.native`, `operators.spark`, `acceleration.ratio`), so the skew is not confined to `EXPLAIN` output — it also biases the acceleration ratio gauge.
Two open design questions for whoever picks this up:
1. For `CometSubqueryBroadcastExec`, mixing in `CometPlan` is the tidier fix but has a wider blast radius, since other rules pattern-match on `CometPlan`. Special-casing it in the explain match is contained but leaves the type lying about itself. Worth checking the other `CometPlan` match sites before choosing.
2. Fixing item 3 means tracking already-visited exchange subtrees during the traversal, which is more involved than items 1 and 2 and could reasonably be split out.
Any of these changes the operator counts, so the TPC-DS plan stability goldens will need regenerating for all supported Spark versions (`./dev/regenerate-golden-files.sh`).
Found while working on #5201, which adds expression-level coverage to the same summary line but does not touch the operator categorization.
Contributor guide
Assessment
This issue has not been assessed yet.