[SUPPORT] Cloudwatch metrics reporter failing to send few data

**Describe the problem you faced**

Hy,

I have an issue with the built-in Cloudwatch Metrics reporter, where it's failing to send some metrics to cloudwatch.
It seems to me as if the session has been stopped before sending the last information.

**To Reproduce**

Run a write operation with metrics enabled to CloudWatch.

**Expected behavior**

No failure, all metrics sent successfully

**Environment Description**

* Hudi version : 0.14.1

* Spark version : 3.5

**Stacktrace**

```
25/04/15 08:30:14 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
25/04/15 08:30:14 INFO SparkContext: Invoking stop() from shutdown hook
25/04/15 08:30:14 INFO SparkContext: SparkContext is stopping with exitCode 0.
25/04/15 08:30:14 INFO SparkUI: Stopped Spark web UI at <>
25/04/15 08:30:14 ERROR CloudWatchReporter: Error reporting metrics to CloudWatch. The data in this CloudWatch request may have been discarded, and not made it to CloudWatch.
java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: event executor terminated
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_432]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[?:1.8.0_432]
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:234) ~[hudi-aws_0.14.1-HudiMetricsReporter-1.0.jar:?]
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:211) ~[hudi-aws_0.14.1-HudiMetricsReporter-1.0.jar:?]
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237) ~[hudi-utilities-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at org.apache.hudi.metrics.cloudwatch.CloudWatchMetricsReporter.report(CloudWatchMetricsReporter.java:71) ~[hudi-utilities-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_432]
at org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:118) ~[hudi-utilities-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_432]
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: event executor terminated
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_432]
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792) ~[?:1.8.0_432]
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153) ~[?:1.8.0_432]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.attemptExecute(AsyncRetryableStage.java:156) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:136) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.execute(AsyncRetryableStage.java:95) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:79) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:44) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:41) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:29) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:64) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:36) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:49) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:32) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonAsyncHttpClient.java:215) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.invoke(BaseAsyncClientHandler.java:288) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:227) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$1(BaseAsyncClientHandler.java:80) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:294) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:73) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:49) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.services.cloudwatch.DefaultCloudWatchAsyncClient.putMetricData(DefaultCloudWatchAsyncClient.java:2601) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:229) ~[hudi-aws_0.14.1-HudiMetricsReporter-1.0.jar:?]
... 6 more
Caused by: java.util.concurrent.RejectedExecutionException: event executor terminated
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.lazyExecute0(SingleThreadEventExecutor.java:831) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.lazyExecute(SingleThreadEventExecutor.java:822) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractScheduledEventExecutor.schedule(AbstractScheduledEventExecutor.java:292) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractScheduledEventExecutor.schedule(AbstractScheduledEventExecutor.java:205) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractEventExecutorGroup.schedule(AbstractEventExecutorGroup.java:50) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.DelegatingEventLoopGroup.schedule(DelegatingEventLoopGroup.java:153) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.acquire(HealthCheckedChannelPool.java:67) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.CancellableAcquireChannelPool.acquire(CancellableAcquireChannelPool.java:50) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.SimpleChannelPoolAwareChannelPool.acquire(SimpleChannelPoolAwareChannelPool.java:42) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.acquireChannel(NettyRequestExecutor.java:104) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.execute(NettyRequestExecutor.java:93) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient.execute(NettyNioAsyncHttpClient.java:123) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.doExecuteHttpRequest(MakeAsyncHttpRequestStage.java:204) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.executeHttpRequest(MakeAsyncHttpRequestStage.java:151) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$1(MakeAsyncHttpRequestStage.java:104) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_432]
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_432]
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_432]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(MakeAsyncHttpRequestStage.java:100) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(MakeAsyncHttpRequestStage.java:65) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallAttemptMetricCollectionStage.execute(AsyncApiCallAttemptMetricCollectionStage.java:62) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallAttemptMetricCollectionStage.execute(AsyncApiCallAttemptMetricCollectionStage.java:41) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.attemptExecute(AsyncRetryableStage.java:144) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:136) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.execute(AsyncRetryableStage.java:95) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:79) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:44) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:41) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:29) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:64) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:36) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:49) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:32) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonAsyncHttpClient.java:215) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.invoke(BaseAsyncClientHandler.java:288) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:227) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$1(BaseAsyncClientHandler.java:80) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:294) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:73) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:49) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at software.amazon.awssdk.services.cloudwatch.DefaultCloudWatchAsyncClient.putMetricData(DefaultCloudWatchAsyncClient.java:2601) ~[aws-sdk-java-bundle-2.23.18.jar:?]
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:229) ~[hudi-aws_0.14.1-HudiMetricsReporter-1.0.jar:?]
... 6 more
25/04/15 08:30:14 INFO YarnClusterSchedulerBackend: Shutting down all executors
25/04/15 08:30:14 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
```

i tried reducing the report interval and then i got the following error:
```
25/05/16 10:02:37 WARN Metrics: Error while closing reporter
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@abaa245 rejected from java.util.concurrent.ThreadPoolExecutor@47d4d62[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_432]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_432]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_432]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_432]
at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient.putMetricDataAsync(AmazonCloudWatchAsyncClient.java:1374) ~[aws-java-sdk-bundle-1.12.705.jar:?]
at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient.putMetricDataAsync(AmazonCloudWatchAsyncClient.java:1366) ~[aws-java-sdk-bundle-1.12.705.jar:?]
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:229) ~[hudi-aws_0.13.1-HudiMetricsReporter-1.0.jar:?]
at org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:212) ~[hudi-aws_0.13.1-HudiMetricsReporter-1.0.jar:?]
at org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237) ~[hudi-spark3.5-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at org.apache.hudi.metrics.cloudwatch.CloudWatchMetricsReporter.report(CloudWatchMetricsReporter.java::55) ~[hudi-aws_0.13.1-HudiMetricsReporter-1.0.jar:?]
at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_432]
at org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:118) ~[hudi-spark3.5-bundle_2.12-0.14.1-amzn-1.jar:0.14.1-amzn-1]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_432]
```

Start with org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report at the stack-trace locations 211, 229, and 234, then trace shutdown through CloudWatchMetricsReporter.report and Metrics.shutdown. Reproduce a write operation with CloudWatch metrics enabled and observe the shutdown sequence. Done means the final metrics are sent successfully without the event executor terminated error.

Written by the indexing model from the issue text.