[Bug] ShutdownHook was not called when SparkSQLEngine exited gracefully

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

SparkSQLEngine did not call ShutdownHook when it was launched in connection share level and stopped gracefully when session was closed.

This problem appeared in SparkSQLEngine launched in spark deploy mode local and yarn-client. Other spark deploy mode has not been checked.

### Affects Version(s)

1.6.0/1.6.1

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

```logtalk
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLSessionManager: Session stopped due to shared level is Connection.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
23/02/09 11:56:51 WARN [main-EventThread] ZookeeperDiscoveryClient: This Kyuubi instance 10.221.99.69:50439 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] EngineServiceDiscovery: Clean up discovery service due to this is connection share level.
23/02/09 11:56:51 INFO [Curator-Framework-0] CuratorFrameworkImpl: backgroundOperationsLoop exiting
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] ZooKeeper: Session: 0x10004e31d840004 closed
23/02/09 11:56:51 INFO [main-EventThread] ClientCnxn: EventThread shut down for session: 0x10004e31d840004
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] EngineServiceDiscovery: Service[EngineServiceDiscovery] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLEngine: Service: [SparkSQLBackendService] is stopping.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLBackendService: Service: [SparkSQLSessionManager] is stopping.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLSessionManager: Service: [SparkSQLOperationManager] is stopping.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLOperationManager: Service[SparkSQLOperationManager] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLSessionManager: Service[SparkSQLSessionManager] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLBackendService: Service[SparkSQLBackendService] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLEngine: Service[SparkSQLEngine] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkTBinaryFrontendService: Finished closing SessionHandle [b98cec81-6444-4803-8431-7dd6dc56a4e5]
23/02/09 11:56:51 INFO [main] SparkUI: Stopped Spark web UI at http://10.221.99.69:50428
```

### Kyuubi Server Configurations

_No response_

### Kyuubi Engine Configurations

_No response_

### Additional context

If SparkSQLEngine was killed by command, or was running in other share level, ShutdownHook was called.

```
23/02/09 11:58:26 ERROR [SIGTERM handler] SparkSQLEngine: RECEIVED SIGNAL 15: TERM
23/02/09 11:58:26 INFO [Thread-1] SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
23/02/09 11:58:26 INFO [Thread-1] SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
23/02/09 11:58:26 INFO [Thread-1] EngineServiceDiscovery: Clean up discovery service due to this is connection share level.
23/02/09 11:58:26 WARN [main-EventThread] ZookeeperDiscoveryClient: This Kyuubi instance 10.221.99.69:50499 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
23/02/09 11:58:26 INFO [Curator-Framework-0] CuratorFrameworkImpl: backgroundOperationsLoop exiting
23/02/09 11:58:26 INFO [main-EventThread] EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
23/02/09 11:58:26 INFO [Thread-1] ZooKeeper: Session: 0x10004e31d840006 closed
23/02/09 11:58:26 INFO [Thread-1] EngineServiceDiscovery: Service[EngineServiceDiscovery] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped
23/02/09 11:58:26 INFO [Thread-1] SparkSQLEngine: Service: [SparkSQLBackendService] is stopping.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLBackendService: Service: [SparkSQLSessionManager] is stopping.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLSessionManager: Service: [SparkSQLOperationManager] is stopping.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLOperationManager: Service[SparkSQLOperationManager] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLSessionManager: Service[SparkSQLSessionManager] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLBackendService: Service[SparkSQLBackendService] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLEngine: Service[SparkSQLEngine] is stopped.
23/02/09 11:58:26 INFO [Thread-1] DiskBlockManager: Shutdown hook called
23/02/09 11:58:26 INFO [main] SparkUI: Stopped Spark web UI at http://10.221.99.69:50487
23/02/09 11:58:26 INFO [Thread-1] ShutdownHookManager: Shutdown hook called
23/02/09 11:58:26 INFO [Thread-1] ShutdownHookManager: Deleting directory /private/var/folders/yj/25xqj6_52n51xmctftgl_77c0000gn/T/spark-92945c05-f7e3-4fae-86fe-f1a409f141fa
23/02/09 11:58:26 INFO [Thread-1] ShutdownHookManager: Deleting directory /private/var/folders/yj/25xqj6_52n51xmctftgl_77c0000gn/T/spark-dd00d8b3-d228-4750-83dd-191059e842a7
23/02/09 11:58:26 INFO [Thread-1] ShutdownHookManager: Deleting directory /private/var/folders/yj/25xqj6_52n51xmctftgl_77c0000gn/T/spark-92945c05-f7e3-4fae-86fe-f1a409f141fa/userFiles-676192b3-9da0-4bd0-8814-84d82abd1ad1
23/02/09 11:58:26 INFO [dispatcher-event-loop-3] MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
```

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Start with the SparkSQLEngine shutdown path and the SparkSQLSessionManager behavior when a connection-share session closes. Compare graceful shutdown with the SIGTERM sequence in the logs, focusing on when ShutdownHook is invoked. Done means graceful exits in the affected deploy modes invoke the shutdown hook consistently.

Written by the indexing model from the issue text.