apache / apache/gluten

[Qualification]Operator Count Inflation in UnsupportedOperators Report

Open
#11,394 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

### Bug description

The `UnsupportedOperators.tsv` report shows inflated operator counts that don't match the actual number of operator instances in the execution plan.

- Count values are 3-15x higher than actual operator instances
- Makes the report misleading for users
- CPU time metrics are NOT affected (still correct)

### Example

For this execution plan:
```
Execute InsertIntoHadoopFsRelationCommand (node 0, unsupported #1 - ROOT)
└── Project (node 1)
└── SortMergeJoin(skew=true) (node 2, unsupported #2)
├── Scan table1 (node 3)
└── Scan table2 (node 4)
```

**Expected**: 2 operators (1 InsertIntoHadoopFsRelationCommand + 1 SortMergeJoin)
**Actual**: 4 in the report

In production event logs, the issue is more severe:
- `Execute InsertIntoHadoopFsRelationCommand`: reported 62, should be 4
- `SortMergeJoin(skew=true)`: reported 62, should be 4

### Root Cause

In `ResultVisitor.java`, the `increment()` method is called for every parent node during post-order traversal:

```java
// Current (wrong) behavior:
currentNodeUnsupportedOperators.forEach(id -> {
currentCost.addCpuDuration(durationMetric.getDuration());
currentCost.increment(); // ❌ Called for each parent node
});
```

When traversing node 1 (Project), it inherits unsupported operator from node 2 (SortMergeJoin) and increments the count again. Same for node 0, causing the count to be 3 instead of 1 for the SortMergeJoin.

### Solution

Only count when the operator is first encountered:

```java
// Fixed behavior:
if (isOperatorNotSupported(nodeId)) {
currentNodeUnsupportedOperators.add(nodeId);
unsupportedOperatorImpactCostMap.put(nodeId, new UnsupportedImpact());
unsupportedOperatorImpactCostMap.get(nodeId).increment(); // ✅ Count once
}

// When visiting parent nodes, only accumulate CPU time:
currentNodeUnsupportedOperators.forEach(id -> {
currentCost.addCpuDuration(durationMetric.getDuration());
// No increment() here
});
```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start in ResultVisitor.java and trace the post-order traversal that updates unsupported-operator costs. Reproduce the shown execution-plan or event-log case and verify that UnsupportedOperators.tsv counts each unsupported operator instance once while CPU time metrics remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.