apache / apache/kyuubi

[Bug] Spark SQL engine main thread can wait forever after SparkContext stops

Open
#7,590 2 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the issues and found related issue #7163.

### Describe the bug

This is a follow-up to #7163.

When an uncaught error such as an `OutOfMemoryError` stops the `SparkContext`, an engine shutdown can start on another thread but stall before `Serverable.stopServer()` is reached. For example, a service may block while releasing resources from an already memory-constrained process.

`SparkSQLEngine.main` waits only on `countDownLatch`. The latch is released by `stopServer()`, so if shutdown stalls before that point, the engine main thread waits forever even though its `SparkContext` has already stopped. In YARN cluster mode this can leave the application master process alive after the driver RPC endpoint has gone away.

#7167 makes the session terminating checker call `stop()` after detecting a stopped `SparkContext`, but it cannot unblock the main thread when that `stop()` call itself stalls.

The engine main thread should also observe the `SparkContext` state and stop waiting when the context has already stopped.

### Affects Version(s)

master

### Kyuubi Server Log Output

Not applicable.

### Kyuubi Engine Log Output

The failure starts with an uncaught error stopping the SparkContext. Engine shutdown then begins but does not reach the final `SparkSQLEngine` stopped message, while the process remains alive.

### Kyuubi Server Configurations

No special configuration is required.

### Kyuubi Engine Configurations

No special configuration is required.

### Additional context

Related public reports:

- #7163
- #7309

### Are you willing to submit PR?

- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.

Contributor guide

Open the contributing guide

Research direction

Start at SparkSQLEngine.main and trace how countDownLatch, SparkContext state, and Serverable.stopServer() interact during shutdown. Review the session terminating checker and related issue context, then verify that the engine main thread no longer waits forever after SparkContext has stopped, including when shutdown stalls before stopServer().

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.