Shutdown hooks cannot be modified during shutdown thrown during plugin intialization
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
[Expected behavior] and [actual behavior].
We have seen cases where Spark shuts down due to some exception while Gluten plugin is being initialized, Gluten plugin tries to register shutdown hooks for cleaning up temporary directories. However, adding new shutdown hooks once the shutdown sequence has started is not allowed leading to an IllegalStateException and preventing graceful shutdown.
### Gluten version
Gluten-1.3
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.
at org.apache.spark.util.SparkShutdownHookManager.add(ShutdownHookManager.scala:195)
at org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:153)
at org.apache.spark.util.SparkShutdownManagerUtil$.addHookForTempDirRemoval(SparkShutdownManagerUtil.scala:29)
at org.apache.spark.util.SparkDirectoryUtil.$anonfun$ROOTS$1(SparkDirectoryUtil.scala:40)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
at org.apache.spark.util.SparkDirectoryUtil.(SparkDirectoryUtil.scala:35)
at org.apache.spark.util.SparkDirectoryUtil$.init(SparkDirectoryUtil.scala:79)
at org.apache.spark.util.SparkDirectoryUtil$.init(SparkDirectoryUtil.scala:74)
at org.apache.gluten.backendsapi.velox.VeloxListenerApi.onDriverStart(VeloxListenerApi.scala:95)
at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart(SubstraitBackend.scala:27)
at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart$(SubstraitBackend.scala:26)
at org.apache.gluten.backendsapi.velox.VeloxBackend.onDriverStart(VeloxBackend.scala:62)
at org.apache.gluten.GlutenDriverPlugin.$anonfun$init$1(GlutenPlugin.scala:72)
at org.apache.gluten.GlutenDriverPlugin.$anonfun$init$1$adapted(GlutenPlugin.scala:72)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.gluten.GlutenDriverPlugin.init(GlutenPlugin.scala:72)
at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$driverPlugins$1(PluginContainer.scala:53)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at org.apache.spark.internal.plugin.DriverPluginContainer.(PluginContainer.scala:46)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:210)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:193)
```
Contributor guide
Research direction
Start with SparkDirectoryUtil.scala and SparkShutdownManagerUtil.scala, following the stack trace from VeloxListenerApi.onDriverStart in the Gluten plugin initialization path. Reproduce the shutdown-time initialization failure and verify that cleanup registration no longer throws IllegalStateException, allowing graceful shutdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100