Version 1.6.0 reports a duplicated rule error
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
Parameters were:
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=16g
spark.executor.memoryOverhead=1g
spark.task.cpus=1
spark.executor.extraJavaOptions='-XX:+UseG1GC -XX:ActiveProcessorCount=8'
spark.plugins org.apache.gluten.GlutenPlugin
spark.shuffle.manager org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.gluten.loadLibFromJar false
spark.gluten.sql.columnar.backend.lib velox
spark.driver.extraClassPath /opt/gluten-velox/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.6.0.jar
spark.executor.extraClassPath /opt/gluten-velox/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.6.0.jar
Error during execution:
26/05/26 16:41:04 ERROR TaskResources: Task -1 failed with error:
org.apache.gluten.exception.GlutenException: duplicated rule
at org.apache.gluten.vectorized.PlanEvaluatorJniWrapper.nativeValidateWithFailureReason(Native Method)
at org.apache.gluten.vectorized.NativePlanEvaluator.doNativeValidateWithFailureReason(NativePlanEvaluator.java:59)
at org.apache.gluten.backendsapi.velox.VeloxValidatorApi.$anonfun$doNativeValidateWithFailureReason$1(VeloxValidatorApi.scala:50)
at org.apache.spark.task.TaskResources$.runUnsafe(TaskResources.scala:100)
at org.apache.gluten.backendsapi.velox.VeloxValidatorApi.doNativeValidateWithFailureReason(VeloxValidatorApi.scala:48)
at org.apache.gluten.execution.TransformSupport.doNativeValidation(WholeStageTransformer.scala:94)
at org.apache.gluten.execution.TransformSupport.doNativeValidation$(WholeStageTransformer.scala:90)
at org.apache.gluten.execution.FilterExecTransformerBase.doNativeValidation(BasicPhysicalOperatorTransformerala:40)
at org.apache.gluten.execution.FilterExecTransformerBase.doValidateInternal(BasicPhysicalOperatorTransformer.scala:143)
at org.apache.gluten.execution.ValidatablePlan.$anonfun$doValidate$3(ValidatablePlan.scala:104)
at org.apache.gluten.execution.ValidatablePlan.failValidationWithException(ValidatablePlan.scala:53)
at org.apache.gluten.execution.ValidatablePlan.failValidationWithException$(ValidatablePlan.scala:43)
at org.apache.gluten.execution.FilterExecTransformerBase.failValidationWithException(BasicPhysicalOperatorTransformer.scala:40)
at org.apache.gluten.execution.ValidatablePlan.doValidate(ValidatablePlan.scala:106)
at org.apache.gluten.execution.ValidatablePlan.doValidate$(ValidatablePlan.scala:70)
at org.apache.gluten.execution.FilterExecTransformerBase.doValidate(BasicPhysicalOperatorTransformer.scala:40)
at org.apache.gluten.extension.columnar.validator.Validators$FallbackByNativeValidation.validate(Validators.scala:246)
at org.apache.gluten.extension.columnar.validator.Validator$Builder$ValidatorPipeline.$anonfun$validate$1(Validator.scala:90)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91)
at org.apache.gluten.extension.columnar.validator.Validator$Builder$ValidatorPipeline.validate(Validator.scala:87)
at org.apache.gluten.extension.columnar.heuristic.AddFallbackTags.addFallbackTag(AddFallbackTags.scala:36)
at org.apache.gluten.extension.columnar.heuristic.AddFallbackTags.$anonfun$apply$1(AddFallbackTags.scala:29)
at org.apache.gluten.extension.columnar.heuristic.AddFallbackTags.$anonfun$apply$1$adapted(AddFallbackTags.scala:28)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:244)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:243)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:243)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:243)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:243)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:243)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:243)
at org.apache.gluten.extension.columnar.heuristic.AddFallbackTags.apply(AddFallbackTags.scala:28)
at org.apache.gluten.extension.columnar.heuristic.AddFallbackTags.apply(AddFallbackTags.scala:26)
at org.apache.gluten.extension.columnar.heuristic.HeuristicTransform$WithRewrites.$anonfun$apply$3(HeuristicTransform.scala:116)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91)
at org.apache.gluten.extension.columnar.heuristic.HeuristicTransform$WithRewrites.apply(HeuristicTransform.scala:114)
at org.apache.gluten.extension.columnar.heuristic.HeuristicTransform$WithRewrites.apply(HeuristicTransform.scala:104)
at org.apache.gluten.extension.columnar.heuristic.HeuristicTransform.$anonfun$apply$1(HeuristicTransform.scala:59)
at scala.collection.IndexedSeqOptimized.foldLeft(IndexedSeqOptimized.scala:60)
at scala.collection.IndexedSeqOptimized.foldLeft$(IndexedSeqOptimized.scala:68)
at scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:49)
at org.apache.gluten.extension.columnar.heuristic.HeuristicTransform.apply(HeuristicTransform.scala:57)
at org.apache.gluten.extension.columnar.heuristic.HeuristicTransform.apply(HeuristicTransform.scala:53)
at org.apache.gluten.extension.columnar.LoggedRule.$anonfun$apply$1(LoggedRule.scala:44)
at org.apache.gluten.metrics.GlutenTimeMetric$.withNanoTime(GlutenTimeMetric.scala:41)
at org.apache.gluten.metrics.GlutenTimeMetric$.withMillisTime(GlutenTimeMetric.scala:46)
at org.apache.gluten.metrics.GlutenTimeMetric$.recordMillisTime(GlutenTimeMetric.scala:50)
at org.apache.gluten.extension.columnar.LoggedRule.apply(LoggedRule.scala:44)
at org.apache.gluten.extension.columnar.LoggedRule.apply(LoggedRule.scala:29)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:222)
at scala.collection.IndexedSeqOptimized.foldLeft(IndexedSeqOptimized.scala:60)
at scala.collection.IndexedSeqOptimized.foldLeft$(IndexedSeqOptimized.scala:68)
### Gluten version
main branch, Gluten-1.5, Gluten-1.4
### Spark version
Spark-3.5.x
### Spark configurations
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=16g
spark.executor.memoryOverhead=1g
spark.task.cpus=1
spark.executor.extraJavaOptions='-XX:+UseG1GC -XX:ActiveProcessorCount=8'
spark.plugins org.apache.gluten.GlutenPlugin
spark.shuffle.manager org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.gluten.loadLibFromJar false
spark.gluten.sql.columnar.backend.lib velox
spark.driver.extraClassPath /opt/gluten-velox/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.6.0.jar
spark.executor.extraClassPath /opt/gluten-velox/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.6.0.jar
### System information
jdk-17.0.18
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by reproducing the failure with the reported Spark 3.5.x configuration and inspect the validation path through VeloxValidatorApi.scala, ValidatablePlan.scala, Validators.scala, and AddFallbackTags.scala. Trace which rule is registered twice and add a focused regression test; done means the reported configuration no longer produces the duplicated rule error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100