[Bug] Spark SQL engine main thread can wait forever after SparkContext stops
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the issues and found related issue #7163.
### Describe the bug
This is a follow-up to #7163.
When an uncaught error such as an `OutOfMemoryError` stops the `SparkContext`, an engine shutdown can start on another thread but stall before `Serverable.stopServer()` is reached. For example, a service may block while releasing resources from an already memory-constrained process.
`SparkSQLEngine.main` waits only on `countDownLatch`. The latch is released by `stopServer()`, so if shutdown stalls before that point, the engine main thread waits forever even though its `SparkContext` has already stopped. In YARN cluster mode this can leave the application master process alive after the driver RPC endpoint has gone away.
#7167 makes the session terminating checker call `stop()` after detecting a stopped `SparkContext`, but it cannot unblock the main thread when that `stop()` call itself stalls.
The engine main thread should also observe the `SparkContext` state and stop waiting when the context has already stopped.
### Affects Version(s)
master
### Kyuubi Server Log Output
Not applicable.
### Kyuubi Engine Log Output
The failure starts with an uncaught error stopping the SparkContext. Engine shutdown then begins but does not reach the final `SparkSQLEngine` stopped message, while the process remains alive.
### Kyuubi Server Configurations
No special configuration is required.
### Kyuubi Engine Configurations
No special configuration is required.
### Additional context
Related public reports:
- #7163
- #7309
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
Contributor guide
Research direction
Start at SparkSQLEngine.main and trace how countDownLatch, SparkContext state, and Serverable.stopServer() interact during shutdown. Review the session terminating checker and related issue context, then verify that the engine main thread no longer waits forever after SparkContext has stopped, including when shutdown stalls before stopServer().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100