apache / apache/gluten

Shutdown hooks cannot be modified during shutdown thrown during plugin intialization

Open
#11,001 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

[Expected behavior] and [actual behavior].

We have seen cases where Spark shuts down due to some exception while Gluten plugin is being initialized, Gluten plugin tries to register shutdown hooks for cleaning up temporary directories. However, adding new shutdown hooks once the shutdown sequence has started is not allowed leading to an IllegalStateException and preventing graceful shutdown.

### Gluten version

Gluten-1.3

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash
java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.
at org.apache.spark.util.SparkShutdownHookManager.add(ShutdownHookManager.scala:195)
at org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:153)
at org.apache.spark.util.SparkShutdownManagerUtil$.addHookForTempDirRemoval(SparkShutdownManagerUtil.scala:29)
at org.apache.spark.util.SparkDirectoryUtil.$anonfun$ROOTS$1(SparkDirectoryUtil.scala:40)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
at org.apache.spark.util.SparkDirectoryUtil.(SparkDirectoryUtil.scala:35)
at org.apache.spark.util.SparkDirectoryUtil$.init(SparkDirectoryUtil.scala:79)
at org.apache.spark.util.SparkDirectoryUtil$.init(SparkDirectoryUtil.scala:74)
at org.apache.gluten.backendsapi.velox.VeloxListenerApi.onDriverStart(VeloxListenerApi.scala:95)
at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart(SubstraitBackend.scala:27)
at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart$(SubstraitBackend.scala:26)
at org.apache.gluten.backendsapi.velox.VeloxBackend.onDriverStart(VeloxBackend.scala:62)
at org.apache.gluten.GlutenDriverPlugin.$anonfun$init$1(GlutenPlugin.scala:72)
at org.apache.gluten.GlutenDriverPlugin.$anonfun$init$1$adapted(GlutenPlugin.scala:72)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.gluten.GlutenDriverPlugin.init(GlutenPlugin.scala:72)
at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$driverPlugins$1(PluginContainer.scala:53)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at org.apache.spark.internal.plugin.DriverPluginContainer.(PluginContainer.scala:46)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:210)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:193)
```

Contributor guide

Open the contributing guide

Research direction

Start with SparkDirectoryUtil.scala and SparkShutdownManagerUtil.scala, following the stack trace from VeloxListenerApi.onDriverStart in the Gluten plugin initialization path. Reproduce the shutdown-time initialization failure and verify that cleanup registration no longer throws IllegalStateException, allowing graceful shutdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.