Hudi job hangs forever

**Describe the problem you faced**

I encountered an issue using EMR 6.7 with Hudi Version 0.11.0 where the Hudi upsert job did not terminate gracefully and remained stuck in a running state indefinitely, despite the upsert operation being completed. Seems like issues with the post-commit actions but forcefully terminating the job and retrying succeeded. Below provided are the configurations and stacktrace details.

Hudi Configs
* Hoodie metadata stats are enabled : spark.conf.set("hoodie.metadata.enable","true")
* Writer configs - Map(hoodie.datasource.hive_sync.database -> aws_tst_db, hoodie.parquet.small.file.limit -> 104857600, hoodie.datasource.hive_sync.mode -> hms, hoodie.datasource.hive_sync.support_timestamp -> true, path -> s3://<>/aws_tst_db/o_aws_rates_partitioned_tst, hoodie.datasource.write.precombine.field -> last_update_date, hoodie.datasource.hive_sync.partition_fields -> creation_date_partition, hoodie.datasource.hive_sync.use_jdbc -> false, hoodie.datasource.hive_sync.partition_extractor_class -> org.apache.hudi.hive.MultiPartKeysValueExtractor, hoodie.parquet.max.file.size -> 268435456, hoodie.cleaner.parallelism -> 1500, hoodie.parquet.block.size -> 268435456, hoodie.datasource.hive_sync.table -> o_aws_rates_partitioned_tst, hoodie.datasource.meta_sync.condition.sync -> true, hoodie.index.type -> GLOBAL_BLOOM, hoodie.datasource.write.operation -> upsert, hoodie.datasource.hive_sync.enable -> true, hoodie.datasource.write.recordkey.field -> rate_id, hoodie.table.name -> o_aws_rates_partitioned_tst, hoodie.datasource.write.hive_style_partitioning -> true, hoodie.cleaner.policy -> KEEP_LATEST_COMMITS, hoodie.keep.max.commits -> 15, hoodie.upsert.shuffle.parallelism -> 750, hoodie.cleaner.commits.retained -> 10, hoodie.keep.min.commits -> 14, hoodie.datasource.write.partitionpath.field -> creation_date_partition), Append

**Stacktrace**

```Add the stacktrace of the error.```

* Below is the spark logs

```
24/06/27 07:28:50 INFO Javalin: Stopping Javalin ...
24/06/27 07:29:50 WARN QueuedThreadPool: QueuedThreadPool[qtp1495243767]@591f9bf7{STOPPING,8<=0<=250,i=232,r=-1,q=0}[NO_TRY] Couldn't stop Thread[qtp1495243767-615,5,main]
24/06/27 07:29:50 ERROR Javalin: Javalin failed to stop gracefully
java.util.concurrent.TimeoutException
at org.apache.hudi.org.apache.jetty.util.FutureCallback.get(FutureCallback.java:130)
at org.apache.hudi.org.apache.jetty.util.FutureCallback.get(FutureCallback.java:30)
at org.apache.hudi.org.apache.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
at org.apache.hudi.org.apache.jetty.server.Server.doStop(Server.java:447)
at org.apache.hudi.org.apache.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
at io.javalin.Javalin.stop(Javalin.java:203)
at org.apache.hudi.timeline.service.TimelineService.close(TimelineService.java:337)
at org.apache.hudi.client.embedded.EmbeddedTimelineService.stop(EmbeddedTimelineService.java:137)
at org.apache.hudi.client.BaseHoodieClient.stopEmbeddedServerView(BaseHoodieClient.java:94)
at org.apache.hudi.client.BaseHoodieClient.close(BaseHoodieClient.java:86)
at org.apache.hudi.client.BaseHoodieWriteClient.close(BaseHoodieWriteClient.java:1539)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:712)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:313)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:165)
```

* Below is the exception from executor log

```
24/06/27 07:16:53 INFO Executor: Running task 5223.0 in stage 26.0 (TID 9982)
24/06/27 07:17:09 ERROR PriorityBasedFileSystemView: Got error running preferred function. Trying secondary
org.apache.hudi.exception.HoodieRemoteException: Read timed out
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesFromParams(RemoteHoodieTableFileSystemView.java:241)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesBeforeOrOn(RemoteHoodieTableFileSystemView.java:248)
at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:99)
at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestBaseFilesBeforeOrOn(PriorityBasedFileSystemView.java:144)
at org.apache.hudi.index.HoodieIndexUtils.getLatestBaseFilesForPartition(HoodieIndexUtils.java:69)
```

* Below is the exception from Yarm application log

```
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.storage.BlockManagerMaster.removeBroadcast(BlockM
Caused by: java.lang.RuntimeException: org.apache.spark.SparkException: Could not find BlockManagerEndpoint1.
at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:178)
at org.apache.spark.rpc.netty.Dispatcher.postRemoteMessage(Dispatcher.scala:136)
at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:684)
```

Please check and let me know if you need any additional details. Thanks

The hang passes through HoodieSparkSqlWriter.scala, TimelineService.java, EmbeddedTimelineService.java, and BaseHoodieClient.java; start at commitAndPerformPostOperations and the Javalin shutdown timeout. Reproduce with the supplied EMR 6.7 and Hudi 0.11.0 settings, then verify that the upsert exits without embedded timeline-service or Spark RPC failures.

Written by the indexing model from the issue text.