[VL] Component not found when `mvn test` after `mvn install -P iceberg`
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
Reported by @Yohahaha.
Error:
```
24/12/23 11:54:16 WARN SparkSession: Cannot use org.apache.gluten.extension.GlutenSessionExtensions to configure session extensions.
java.lang.NoClassDefFoundError: org/apache/gluten/execution/OffloadIcebergScan$
at org.apache.gluten.component.VeloxIcebergComponent.injectRules(VeloxIcebergComponent.scala:29)
at org.apache.gluten.extension.GlutenSessionExtensions.$anonfun$apply$6(GlutenSessionExtensions.scala:51)
at org.apache.gluten.extension.GlutenSessionExtensions.$anonfun$apply$6$adapted(GlutenSessionExtensions.scala:51)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.gluten.extension.GlutenSessionExtensions.apply(GlutenSessionExtensions.scala:51)
at org.apache.gluten.extension.GlutenSessionExtensions.apply(GlutenSessionExtensions.scala:26)
at org.apache.spark.sql.SparkSession$.$anonfun$applyExtensions$1(SparkSession.scala:1224)
at org.apache.spark.sql.SparkSession$.$anonfun$applyExtensions$1$adapted(SparkSession.scala:1219)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$applyExtensions(SparkSession.scala:1219)
at org.apache.spark.sql.SparkSession.(SparkSession.scala:106)
at org.apache.spark.sql.SparkSession.(SparkSession.scala:109)
at org.apache.spark.sql.test.TestSparkSession.(TestSQLContext.scala:27)
at org.apache.spark.sql.test.TestSparkSession.(TestSQLContext.scala:30)
at org.apache.spark.sql.test.SharedSparkSessionBase.createSparkSession(SharedSparkSession.scala:102)
at org.apache.spark.sql.test.SharedSparkSessionBase.createSparkSession$(SharedSparkSession.scala:100)
at org.apache.gluten.execution.WholeStageTransformerSuite.createSparkSession(WholeStageTransformerSuite.scala:37)
at org.apache.spark.sql.test.SharedSparkSessionBase.initializeSession(SharedSparkSession.scala:116)
at org.apache.spark.sql.test.SharedSparkSessionBase.initializeSession$(SharedSparkSession.scala:114)
at org.apache.gluten.execution.WholeStageTransformerSuite.initializeSession(WholeStageTransformerSuite.scala:37)
at org.apache.spark.sql.test.SharedSparkSessionBase.beforeAll(SharedSparkSession.scala:124)
at org.apache.spark.sql.test.SharedSparkSessionBase.beforeAll$(SharedSparkSession.scala:123)
at org.apache.gluten.execution.WholeStageTransformerSuite.org$apache$spark$sql$test$SharedSparkSession$$super$beforeAll(WholeStageTransformerSuite.scala:37)
at org.apache.spark.sql.test.SharedSparkSession.beforeAll(SharedSparkSession.scala:46)
at org.apache.spark.sql.test.SharedSparkSession.beforeAll$(SharedSparkSession.scala:44)
at org.apache.gluten.execution.WholeStageTransformerSuite.beforeAll(WholeStageTransformerSuite.scala:67)
at org.apache.gluten.execution.VeloxTPCHTableSupport.beforeAll(VeloxTPCHSuite.scala:55)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:64)
at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178)
at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at org.scalatest.Suite.runNestedSuites(Suite.scala:1223)
at org.scalatest.Suite.runNestedSuites$(Suite.scala:1156)
at org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:30)
at org.scalatest.Suite.run(Suite.scala:1111)
at org.scalatest.Suite.run$(Suite.scala:1096)
at org.scalatest.tools.DiscoverySuite.run(DiscoverySuite.scala:30)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
at org.scalatest.tools.Runner$.main(Runner.scala:775)
at org.scalatest.tools.Runner.main(Runner.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.gluten.execution.OffloadIcebergScan$
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 57 more
```
To repeat:
```
mvn -P backends-velox,spark-3.3,scala-2.12,delta,iceberg,uniffle,celeborn clean install -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dspotless.check.skip=true -DskipTests
mvn -P backends-velox,spark-3.3,scala-2.12 test
```
This is a regression due to https://github.com/apache/incubator-gluten/pull/8192.
Contributor guide
Research direction
Reproduce the failure with the two Maven commands in the issue, then trace the missing org.apache.gluten.execution.OffloadIcebergScan$ reference from VeloxIcebergComponent.injectRules and GlutenSessionExtensions. Compare the profiles and regression identified in pull request 8192; done means the second command runs without the NoClassDefFoundError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- build-system, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100