[Qualification]Operator Count Inflation in UnsupportedOperators Report
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
### Bug description
The `UnsupportedOperators.tsv` report shows inflated operator counts that don't match the actual number of operator instances in the execution plan.
- Count values are 3-15x higher than actual operator instances
- Makes the report misleading for users
- CPU time metrics are NOT affected (still correct)
### Example
For this execution plan:
```
Execute InsertIntoHadoopFsRelationCommand (node 0, unsupported #1 - ROOT)
└── Project (node 1)
└── SortMergeJoin(skew=true) (node 2, unsupported #2)
├── Scan table1 (node 3)
└── Scan table2 (node 4)
```
**Expected**: 2 operators (1 InsertIntoHadoopFsRelationCommand + 1 SortMergeJoin)
**Actual**: 4 in the report
In production event logs, the issue is more severe:
- `Execute InsertIntoHadoopFsRelationCommand`: reported 62, should be 4
- `SortMergeJoin(skew=true)`: reported 62, should be 4
### Root Cause
In `ResultVisitor.java`, the `increment()` method is called for every parent node during post-order traversal:
```java
// Current (wrong) behavior:
currentNodeUnsupportedOperators.forEach(id -> {
currentCost.addCpuDuration(durationMetric.getDuration());
currentCost.increment(); // ❌ Called for each parent node
});
```
When traversing node 1 (Project), it inherits unsupported operator from node 2 (SortMergeJoin) and increments the count again. Same for node 0, causing the count to be 3 instead of 1 for the SortMergeJoin.
### Solution
Only count when the operator is first encountered:
```java
// Fixed behavior:
if (isOperatorNotSupported(nodeId)) {
currentNodeUnsupportedOperators.add(nodeId);
unsupportedOperatorImpactCostMap.put(nodeId, new UnsupportedImpact());
unsupportedOperatorImpactCostMap.get(nodeId).increment(); // ✅ Count once
}
// When visiting parent nodes, only accumulate CPU time:
currentNodeUnsupportedOperators.forEach(id -> {
currentCost.addCpuDuration(durationMetric.getDuration());
// No increment() here
});
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start in ResultVisitor.java and trace the post-order traversal that updates unsupported-operator costs. Reproduce the shown execution-plan or event-log case and verify that UnsupportedOperators.tsv counts each unsupported operator instance once while CPU time metrics remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100