apache / apache/kyuubi

[Bug] Hot Upgrade do not work as expected

Open
#7,016 1 comment 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

The Kyuubi server fails to release the activeSession, preventing the server from shutting down properly.

Details:
We’re utilizing the hot upgrade feature, but noticed that the process doesn’t terminate as expected.
Upon examining the logs and source code, we observed that although ZooKeeper has already released the znode, the activeSessionCount remains greater than zero.

This issue is easily reproducible. Simply run zkCli.sh -server [host] while the server is processing a basic query like show databases;.

We believe this might be a bug on the Kyuubi server side. Could you kindly look into this?

Thank you.

### Affects Version(s)

1.9.2

### Kyuubi Server Log Output

```logtalk
2025-04-08 13:28:53.336 KyuubiSessionManager-exec-pool: Thread-113 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing jpz40462's query[cc906dab-b30a-482b-8f08-a132ee073d7b]: PENDING_STATE -> RUNNING_STATE, statement:
show databases
2025-04-08 13:28:53.521 KyuubiSessionManager-exec-pool: Thread-113 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[cc906dab-b30a-482b-8f08-a132ee073d7b] in FINISHED_STATE
2025-04-08 13:28:53.522 KyuubiSessionManager-exec-pool: Thread-113 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing jpz40462's query[cc906dab-b30a-482b-8f08-a132ee073d7b]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.184 seconds
2025-04-08 13:28:53.795 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing jpz40462's query[cc906dab-b30a-482b-8f08-a132ee073d7b]: FINISHED_STATE -> CLOSED_STATE, time taken: 0.458 seconds
2025-04-08 13:28:53.824 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:CC 90 6D AB B3 0A 48 2B 8F 08 A1 32 EE 07 3D 7B, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side
2025-04-08 13:28:53.845 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [6e8bce61-610e-4e78-8e7e-1f64e5f0fc0d]
2025-04-08 13:28:53.847 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.session.KyuubiSessionManager: jpz40462's KyuubiSessionImpl with SessionHandle [6e8bce61-610e-4e78-8e7e-1f64e5f0fc0d] is closed, current opening sessions 0
2025-04-08 13:28:53.848 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.operation.LaunchEngine: Processing jpz40462's query[dc7a1cb6-3924-4f88-94de-bc94a77f509c]: FINISHED_STATE -> CLOSED_STATE, time taken: 28.092 seconds
2025-04-08 13:28:53.918 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [6e8bce61-610e-4e78-8e7e-1f64e5f0fc0d]
...
2025-04-08 13:32:10.856 main-EventThread INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: 1 connection(s) are active, delay shutdown
2025-04-08 13:32:20.857 main-EventThread INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: 1 connection(s) are active, delay shutdown
2025-04-08 13:32:30.857 main-EventThread INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: 1 connection(s) are active, delay shutdown
...
2025-04-08 14:18:40.969 main-EventThread INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: 1 connection(s) are active, delay shutdown
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with zkCli.sh -server while running `show databases`, then trace KyuubiServiceDiscovery and the activeSessionCount handling shown in the logs. Check the session and connection shutdown path; done means the active session is released and the server shuts down after ZooKeeper releases its znode, without repeated “connections are active” delays.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.