[SUPPORT]When I write HUDI with Flink use asynchronous compaction, I use the --service parameter, but I run into a problem

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?yes

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

A clear and concise description of the problem.

**To Reproduce**

Steps to reproduce the behavior:

" 'connector' = 'hudi',\n" +
" 'path' = 'hdfs://sugon-cluster/home/like/zs_test_lk0119',\n" +
" 'index.type' = 'BUCKET', \n" +
" 'hoodie.bucket.index.hash.field' = 'id', \n" +
" 'hoodie.index.bucket.engine' = 'CONSISTENT_HASHING', \n" +
" 'hoodie.clustering.async.max.commits' = '4', \n" +
" 'clustering.schedule.enabled' = 'true',\n" +
" 'hoodie.bucket.index.max.num.buckets' = '128', \n" +
" 'hoodie.bucket.index.min.num.buckets' = '8', \n" +
" 'hoodie.bucket.index.split.threshold' = '1.5', \n" +
" 'hoodie.bucket.index.merge.threshold' = '0.1', \n" +
" 'hoodie.datasource.write.recordkey.field'='id',\n" +
" 'write.tasks' = '4',\n" +
" 'compaction.tasks' = '4',\n" +
" 'hoodie.parquet.max.file.size'= '10000',"+
" 'hoodie.parquet.small.file.limit'='5000'," +
" 'hoodie.logfile.max.size'='268435456',"+
" 'compaction.async.enabled' = 'false',\n" +
" 'compaction.schedule.enabled' = 'true',\n" +
" 'compaction.delta_commits' = '1',\n" +
" 'write.precombine.field'='csrq',\n" +
" 'compaction.max_memory' = '1000',\n" +
" 'write.insert.deduplicate'= 'true',\n" +
" 'write.operation'= 'upsert',\n" +
" 'table.type' = 'MERGE_ON_READ'

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :0.14.0

* Flink version :1.13.6

* Hive version :

* Hadoop version :3.2.1

* Storage (HDFS/S3/GCS..) :hdfs

* Running on Docker? (yes/no) :no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: org.apache.hudi.exception.HoodieException: Failed to scan metadata
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: Failed to scan metadata
at org.apache.hudi.sink.compact.HoodieFlinkCompactor.start(HoodieFlinkCompactor.java:89)
at org.apache.hudi.sink.compact.HoodieFlinkCompactor.main(HoodieFlinkCompactor.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
... 11 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Failed to scan metadata
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:103)
at org.apache.hudi.sink.compact.HoodieFlinkCompactor.start(HoodieFlinkCompactor.java:87)
... 17 more
Caused by: org.apache.hudi.exception.HoodieException: Failed to scan metadata
at org.apache.hudi.sink.compact.HoodieFlinkCompactor$AsyncCompactionService.lambda$startService$0(HoodieFlinkCompactor.java:209)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieIOException: Failed to scan metadata
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.(HoodieActiveTimeline.java:165)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.(HoodieActiveTimeline.java:155)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.(HoodieActiveTimeline.java:175)
at org.apache.hudi.common.table.HoodieTableMetaClient.reloadActiveTimeline(HoodieTableMetaClient.java:366)
at org.apache.hudi.sink.compact.HoodieFlinkCompactor$AsyncCompactionService.compact(HoodieFlinkCompactor.java:220)
at org.apache.hudi.sink.compact.HoodieFlinkCompactor$AsyncCompactionService.lambda$startService$0(HoodieFlinkCompactor.java:198)
... 4 more
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:475)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1645)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1631)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1048)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1112)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1109)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1119)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.lambda$listStatus$19(HoodieWrapperFileSystem.java:597)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:114)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.listStatus(HoodieWrapperFileSystem.java:596)
at org.apache.hudi.common.table.HoodieTableMetaClient.scanFiles(HoodieTableMetaClient.java:552)
at org.apache.hudi.common.table.HoodieTableMetaClient.scanHoodieInstantsFromFileSystem(HoodieTableMetaClient.java:645)
at org.apache.hudi.common.table.HoodieTableMetaClient.scanHoodieInstantsFromFileSystem(HoodieTableMetaClient.java:628)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.(HoodieActiveTimeline.java:163)
... 9 more

Start at HoodieFlinkCompactor.java, especially the async service entry points around lines 198-220, then trace HoodieActiveTimeline and HoodieTableMetaClient metadata scanning. Reproduce with the supplied Hudi 0.14.0, Flink 1.13.6, HDFS configuration and inspect the Filesystem closed failure; done means the cause is isolated and async compaction no longer fails during metadata scanning.

Written by the indexing model from the issue text.