[VL][CH] Gluten columnar rules + RAS integration
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
To achieve one of the original purpose of ras: to make Gluten's rule simpler, and to make the rule list cleaner, some kind of migrations on our current rule code will be suggested.
The following is what I could assume:
#### RAS=on vs RAS=off
I'd raise a new approach around current entrance of RAS. Will be inclined to having two individual source code files (or folders) for RAS=on and RAS=off, to store the rule list of the each way. By doing this, it would be easier for developers to know what and how RAS actually does.
Worth noting that the migration would not remove rules from `RAS=off` as of now. We only doing code refactors to make `RAS=on` cover more columnar optimizations in `RAS=off`.
#### The gap of CH backend
CH backend doesn't currently support RAS=on. We should eliminate the following gaps as far as I know to make them compatible:
- In CH backend there is some other plan nodes that are coupled with others, for example AFAIK the broadcast exchange doesn't work with C2R / R2C.
There is a PR https://github.com/apache/incubator-gluten/pull/5101 to verify CH backend and TPC-H, we can make it pass asap.
#### Rules to migrate
Ideally we'll migrate most of columnar rules that are not 100% heuristic. The following is a initial list and since I may not be the writer of these rules, so please correct me if I get anything wrong.
1. `FallbackOnANSIMode`
Will keep this in heuristics since it's a global switch.
3. `FallbackMultiCodegens`
- choice 1: In RAS, tune cost model to slightly increase Velox's BHJ to larger than Vanilla Spark's BHJ. Then optimizer will fallback consecutive Gluten BHJs when it sees some.
- choice 2: Keep as heuristic
4. `PlanOneRowRelation`
Will keep this in heuristics since it's a global switch.
5. `FallbackEmptySchemaRelation`
Will keep this in heuristics since it's a global switch.
6. `MergeTwoPhasesHashBaseAggregate` (CH only)
- choice 1: Could add a RAS rule to merge aggregates.
- choice 2: Keep as heuristic
7. Spark rewrite rules: `RewriteIn`, `RewriteMultiChildrenCount`, `RewriteCollect`, `RewriteTypedImperativeAggregate`, `PullOutPreProject`, `PullOutPostProject`:
Move them to RAS. RAS would naturally do such "tentative" transformations better than RBO. RAS could enumerate the possible plans then try to find the one that is executable and have lowest cost. For example, if the transformation yields too many of C2Rs/R2Cs, or just an inexecutable plan, optimizer will not choose it.
8. `AddTransformHintRule`
In RAS, merge this main validation rule together with transformation rule. Which means, RAS should validate the plan before it does each tentative transformation.
9. `FallbackBloomFilterAggIfNeeded`
- choice 1: In RAS, add an independent physical property to indicate if the bloom filter buffer data is vanilla or Gluten/Velox. Then
optimizer would only choose plans that have consistent buffer data format.
- choice 2: Keep as heuristic
10. `ImplementFilter`, `ImplementAggregate`, `ImplementExchange`, `ImplementJoin`, `ImplementOthers`
Can be moved to RAS, to do transformation.
11. `RemoveNativeWriteFilesSortAndProject`
The "remove sort and project" part can be perfectly done by RAS by property requirements. The other `Empty2Null` looks like some kind of validation that can be moved into RAS's transformation rule. Confirmation required.
12. `RewriteTransformer`
RAS could directly use the plugged rules, ideally.
13. `EnsureLocalSortRequirements`
It's RAS's natural job. Though we just need to add ordering property to property model.
14. `CollapseProjectExecTransformer`
RAS can use that rule.
15. `genExtendedColumnarTransformRules` (Velox only)
Currently only flushable agg rule. Can move to RAS's agg transformation rule.
16. `GlutenConfig.getConf.extendedColumnarTransformRules`
Keep it at the end of heuristic rule list to allow user defined rewrites. Will not add to RAS.
17. `ExpandFallbackPolicy` (whole stage fallback)
This can be done by RAS by setting more reasonable costs to C2R / R2C. Then optimizer would consider the cost of a plan with a lot of C2Rs / R2Cs higher then will not choose it.
18. `InsertTransitions`, `TransformPostOverrides`, `InsertColumnarToColumnarTransitions`, `RemoveGlutenTableCacheColumnarToRow`
RAS doesn't need these rules. C2Rs / R2Cs can be added via property enforcement.
19. `RemoveTopmostColumnarToRow`
Since this one is for compatible with Spark, will keep it in heuristics.
20. `genExtendedColumnarPostRules`
Probably keep in heuristics.
21. `ColumnarCollapseTransformStages`
Probably keep in heuristics.
22. `extendedColumnarRules`
Keep in heuristics.
23. `GlutenFallbackReporter`
Keep in heuristics.
24. `RemoveTransformHintRule`
Keep in heuristics.
#### Removal of RAS=off
This can be one of the final goal if all goes well as expected. But we will not take this action in short term.
#### Note
This topic would not only be related to code quality and maintenance. After we have a cleaner rule list and plan node definitions that are fully compatible to RAS, then we can decide whether to move on from RAS's current responsibility (fallback processing) to more advanced optimizations that could be powered by RAS.
Contributor guide
Research direction
Start by reviewing PR 5101 and the RAS entry points and rule names listed in the issue, especially the ClickHouse backend compatibility gaps. Map which rules should remain heuristic and which could move into RAS, including the property and transition concerns described. Done means an agreed migration scope and verified RAS-on compatibility for the affected backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala, spark
- Domain
- backend, data-engineering, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100