apache / apache/gluten

Simplify StrictRule due to transform up

Open
#10,559 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

The implementation of StrictRule used to replaces the children with `DummyLeafExec` s so they become inaccessible afterward.

But there exists a conflict that `LegacyOffload` applies each rule with `transformUp`.
```
class LegacyOffload(rules: Seq[OffloadSingleNode]) extends Rule[SparkPlan] with LogLevelUtil {
def apply(plan: SparkPlan): SparkPlan = {
val out =
rules.foldLeft(plan)((p, rule) => p.transformUp { case p => rule.offload(p) })
out
}
}
```
In other words, the class `DummyLeafExec` is invalid due to the mechanism of `transformUp`.

Take an example, a `ProjectExec` has a child `FileSourceScanExec`. The `StrictRule` will transform `FileSourceScanExec` to `FileSourceScanExecTransformer` and then transform `ProjectExec` to `ProjectExecTransformer` due to `transformUp`.

When hiding the children of `ProjectExec`, the child of `ProjectExec` already be the `ProjectExecTransformer`, so `StrictRule` can't hide anything.

### Gluten version

main branch

Contributor guide

Open the contributing guide

Research direction

Start by reading the StrictRule and LegacyOffload implementations and tracing how transformUp visits SparkPlan nodes. Confirm how ProjectExec and FileSourceScanExec are transformed, then simplify StrictRule so child hiding is no longer invalid under transformUp; done means the offload transformations retain their intended behavior without relying on DummyLeafExec.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.