Simplify StrictRule due to transform up
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
The implementation of StrictRule used to replaces the children with `DummyLeafExec` s so they become inaccessible afterward.
But there exists a conflict that `LegacyOffload` applies each rule with `transformUp`.
```
class LegacyOffload(rules: Seq[OffloadSingleNode]) extends Rule[SparkPlan] with LogLevelUtil {
def apply(plan: SparkPlan): SparkPlan = {
val out =
rules.foldLeft(plan)((p, rule) => p.transformUp { case p => rule.offload(p) })
out
}
}
```
In other words, the class `DummyLeafExec` is invalid due to the mechanism of `transformUp`.
Take an example, a `ProjectExec` has a child `FileSourceScanExec`. The `StrictRule` will transform `FileSourceScanExec` to `FileSourceScanExecTransformer` and then transform `ProjectExec` to `ProjectExecTransformer` due to `transformUp`.
When hiding the children of `ProjectExec`, the child of `ProjectExec` already be the `ProjectExecTransformer`, so `StrictRule` can't hide anything.
### Gluten version
main branch
Contributor guide
Research direction
Start by reading the StrictRule and LegacyOffload implementations and tracing how transformUp visits SparkPlan nodes. Confirm how ProjectExec and FileSourceScanExec are transformed, then simplify StrictRule so child hiding is no longer invalid under transformUp; done means the offload transformations retain their intended behavior without relying on DummyLeafExec.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100