[CH] Make sure that all processors are cancelled before releasing global resources in native engine and after yarn kill or driver shutdown command comes

### Backend

CH (ClickHouse)

### Bug description


### Spark version
3.3

### Spark configurations

AQE enabled

### System information

_No response_

### Relevant logs

```bash
2024/05/17 03:41:06,432 INFO [Executor task launch worker for task 15.0 in stage 1.0 (TID 67)] Executor: Finished task 15.0 in stage 1.0 (TID 67). 5183 bytes result sent to driver
2024/05/17 03:41:11,329 INFO [dispatcher-Executor] YarnCoarseGrainedExecutorBackend: Driver commanded a shutdown
2024/05/17 03:41:11,376 INFO [CoarseGrainedExecutorBackend-stop-executor] MemoryStore: MemoryStore cleared
2024/05/17 03:41:11,376 INFO [CoarseGrainedExecutorBackend-stop-executor] BlockManager: BlockManager stopped
2024-05-17 03:41:11.381 jni: start nativeOnTerminate
2024/05/17 03:41:12,445 INFO [shutdown-hook-0] JniLibLoader: Start unload library path /data17/hadoop/yarn/local/filecache/1281177/libch_240516.so
2024/05/17 03:41:12,446 INFO [shutdown-hook-0] JniLibLoader: verbosePath: /usr/hdp/3.1.0.0-78/hadoop/lib/native/libhadoop.so.1.0.0, libPath: /data17/hadoop/yarn/local/filecache/1281177/libch_240516.so
2024/05/17 03:41:12,446 INFO [shutdown-hook-0] JniLibLoader: Skip finalize because libhadoop.so.1.0.0 != libch_240516.so
2024/05/17 03:41:12,446 INFO [shutdown-hook-0] JniLibLoader: verbosePath: /data17/hadoop/yarn/local/filecache/1281177/libch_240516.so, libPath: /data17/hadoop/yarn/local/filecache/1281177/libch_240516.so
2024/05/17 03:41:12,446 INFO [shutdown-hook-0] JniLibLoader: Start finalize library file libch_240516.so
2024-05-17 03:41:12.446 jni: start JNI_OnUnload
2024/05/17 03:41:12,446 INFO [shutdown-hook-0] JniLibLoader: verbosePath: /data5/hadoop/yarn/local/usercache/xumingyong/appcache/application_1710462985812_773900/container_e57_1710462985812_773900_01_000004/tmp/liblz4-java-1735226946575805094.so, libPath: /data17/hadoop/yarn/local/filecache/1281177/libch_240516.so
2024/05/17 03:41:12,447 INFO [shutdown-hook-0] JniLibLoader: Skip finalize because liblz4-java-1735226946575805094.so != libch_240516.so
2024/05/17 03:41:12,456 INFO [shutdown-hook-0] ShutdownHookManager: Shutdown hook called
```
```

Start from the shutdown sequence shown in the logs, especially nativeOnTerminate and JNI_OnUnload, and trace how processor cancellation and global-resource release are ordered for YARN kill and driver shutdown. Reproduce the shutdown path with Spark 3.3 and AQE enabled, then verify that all processors are cancelled before native resources and the library are released.

Written by the indexing model from the issue text.