[Bug][Spark] Spark SQL engine shutdown fails when no active Spark session is available
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
I encountered an unexpected exception:
```
26/08/11 23:53:47 ERROR AsyncEventQueue: Listener SparkSQLEngineListener threw an exception
org.apache.spark.SparkException: [INTERNAL_ERROR] No active or default Spark session found SQLSTATE: XX000
at org.apache.spark.SparkException$.internalError(SparkException.scala:104)
at org.apache.spark.SparkException$.internalError(SparkException.scala:108)
at org.apache.spark.sql.SparkSessionCompanion.$anonfun$active$2(SparkSession.scala:1037)
at scala.Option.getOrElse(Option.scala:201)
at org.apache.spark.sql.SparkSessionCompanion.$anonfun$active$1(SparkSession.scala:1037)
at scala.Option.getOrElse(Option.scala:201)
at org.apache.spark.sql.SparkSessionCompanion.active(SparkSession.scala:1036)
at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.globalSparkContext(KyuubiSparkUtil.scala:41)
at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.engineId(KyuubiSparkUtil.scala:58)
at org.apache.kyuubi.engine.spark.session.SparkSQLSessionManager.getEngineResultSavePath(SparkSQLSessionManager.scala:218)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineSavePath$lzycompute(SparkSQLEngine.scala:62)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineSavePath(SparkSQLEngine.scala:61)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.$anonfun$stop$3(SparkSQLEngine.scala:119)
at org.apache.kyuubi.Logging.error(Logging.scala:88)
at org.apache.kyuubi.Logging.error$(Logging.scala:86)
at org.apache.kyuubi.service.AbstractService.error(AbstractService.scala:23)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.stop(SparkSQLEngine.scala:119)
at org.apache.spark.kyuubi.SparkSQLEngineListener.onApplicationEnd(SparkSQLEngineListener.scala:61)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:57)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:38)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:38)
at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:118)
at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:102)
at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:106)
at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:106)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:101)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:97)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1293)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:97)
```
### Affects Version(s)
1.12.0
### Kyuubi Server Log Output
```logtalk
```
### Kyuubi Engine Log Output
```logtalk
```
### Kyuubi Server Configurations
```yaml
```
### Kyuubi Engine Configurations
```yaml
```
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Assessment
This issue has not been assessed yet.