File not found while using metadata table for insert_overwrite table

We are incrementally writing to a hudi table with insert_overwrite operations. Recently, We enabled Hudi metadata table for these tables. However after few days we started to encounter the `FileNotFoundException` issue while reading these tables from athena (with metadata listing enabled). Upon further investigation, we observed that the metadata contained older files references that were cleaned up by the cleaner and are no longer available.

Steps to reproduce the behavior:
1. create a simple df and write to a hudi table incrementally with these properties
```
hoodie.datasource.meta.sync.enable=true
hoodie.meta.sync.client.tool.class=org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool
hoodie.write.markers.type=DIRECT
**hoodie.metadata.enable=true
hoodie.datasource.write.operation=insert_overwrite**
hoodie.datasource.write.partitionpath.field=cs_load_hr
hoodie.datasource.hive_sync.partition_fields=cs_load_hr
partition.assignment.strategy=org.apache.kafka.clients.consumer.RangeAssignor
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.TimestampBasedKeyGenerator
hoodie.deltastreamer.keygen.timebased.timestamp.type=DATE_STRING
hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd/HH
hoodie.deltastreamer.source.hoodieincr.partition.extractor.class=org.apache.hudi.hive.SlashEncodedHourPartitionValueExtractor
hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.SlashEncodedHourPartitionValueExtractor
hoodie.parquet.compression.codec=snappy
hoodie.table.services.enabled=true
hoodie.rollback.using.markers=false
hoodie.commits.archival.batch=30
hoodie.archive.delete.parallelism=500
hoodie.index.type=SIMPLE
hoodie.clean.allow.multiple=false
hoodie.clean.async=true
hoodie.clean.automatic=true
hoodie.cleaner.policy=KEEP_LATEST_COMMITS
hoodie.cleaner.commits.retained=3
hoodie.cleaner.parallelism=500
hoodie.cleaner.incremental.mode=true
hoodie.clean.max.commits=8
hoodie.archive.async=true
hoodie.archive.automatic=true
hoodie.archive.merge.enable=true
hoodie.archive.merge.files.batch.size=60
hoodie.keep.max.commits=10
hoodie.keep.min.commits=5
```
df.write.format("org.apache.hudi").options(hudiOptions).mode(SaveMode.Append).save(hudiOutputTablePath)

2. after few incremental writes, some of the base files should be updated & metadata does not get updated properly, it continues to persist old files pointer as well.
3. if you try reading the table using spark or athena, you will get FileNotFoundException
keep in mind to enable metadata while reading. upon disabling the metadata listing on the read side, there is no error and reads work fine.
4. Note : We have observed this issue only for **insert_overwrite** operations. Upsert operation table's metadata gets updated correctly.

**Expected behavior**

It is expected that the hoodie metadata gets updated correctly.

**Environment Description**

* Hudi version : 0.13.1

* Spark version : 3.2.1

* Hive version : NA

* Hadoop version :

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) :

**Additional context**

The timeline also contains replaceCommits for corrupted tables. (which are not present in case of upsert table)

```
$ aws s3 ls s3://tmp-data/investments_ctr_tbl/.hoodie/
PRE .aux/
PRE archived/
PRE metadata/
2023-12-08 13:32:17 0 .aux_$folder$
2023-12-08 13:32:17 0 .schema_$folder$
2023-12-08 13:32:17 0 .temp_$folder$
2023-12-14 22:17:18 4678 20231214221641350.clean
2023-12-14 22:17:11 3227 20231214221641350.clean.inflight
2023-12-14 22:17:10 3227 20231214221641350.clean.requested
2023-12-22 21:50:54 4439 20231222214849300.clean
2023-12-22 21:50:45 4337 20231222214849300.clean.inflight
2023-12-22 21:50:45 4337 20231222214849300.clean.requested
2023-12-30 21:51:16 4439 20231230214431936.clean
2023-12-30 21:51:07 4337 20231230214431936.clean.inflight
2023-12-30 21:51:07 4337 20231230214431936.clean.requested
2024-01-07 21:53:30 4439 20240107215204594.clean
2024-01-07 21:53:23 4337 20240107215204594.clean.inflight
2024-01-07 21:53:22 4337 20240107215204594.clean.requested
2024-01-15 21:55:00 4439 20240115215112126.clean
2024-01-15 21:54:52 4337 20240115215112126.clean.inflight
2024-01-15 21:54:52 4337 20240115215112126.clean.requested
2024-01-23 21:46:53 4439 20240123214442067.clean
2024-01-23 21:46:45 4337 20240123214442067.clean.inflight
2024-01-23 21:46:45 4337 20240123214442067.clean.requested
2024-01-27 21:42:46 4107 20240127214137934.replacecommit
2024-01-27 21:42:22 2221 20240127214137934.replacecommit.inflight
2024-01-27 21:41:42 2522 20240127214137934.replacecommit.requested
2024-01-28 21:47:38 4107 20240128214645441.replacecommit
2024-01-28 21:47:15 2221 20240128214645441.replacecommit.inflight
2024-01-28 21:46:50 2522 20240128214645441.replacecommit.requested
2024-01-29 21:48:52 4107 20240129214748785.replacecommit
2024-01-29 21:48:23 2221 20240129214748785.replacecommit.inflight
2024-01-29 21:47:55 2522 20240129214748785.replacecommit.requested
2024-01-30 21:47:51 4107 20240130214652235.replacecommit
2024-01-30 21:47:26 2221 20240130214652235.replacecommit.inflight
2024-01-30 21:46:58 2522 20240130214652235.replacecommit.requested
2024-01-31 21:46:06 4107 20240131214409159.replacecommit
2024-01-31 21:45:47 2221 20240131214409159.replacecommit.inflight
2024-01-31 21:44:16 2522 20240131214409159.replacecommit.requested
2024-01-31 21:45:57 4439 20240131214417071.clean
2024-01-31 21:45:49 4337 20240131214417071.clean.inflight
2024-01-31 21:45:49 4337 20240131214417071.clean.requested
2024-02-01 21:46:04 4107 20240201214350334.replacecommit
2024-02-01 21:45:35 2221 20240201214350334.replacecommit.inflight
2024-02-01 21:43:59 2522 20240201214350334.replacecommit.requested
2024-02-02 21:48:21 4107 20240202214718803.replacecommit
2024-02-02 21:47:56 2221 20240202214718803.replacecommit.inflight
2024-02-02 21:47:26 2522 20240202214718803.replacecommit.requested
2024-02-03 21:52:09 4107 20240203215112346.replacecommit
2024-02-03 21:51:46 2221 20240203215112346.replacecommit.inflight
2024-02-03 21:51:18 2522 20240203215112346.replacecommit.requested
2024-02-04 21:49:18 4107 20240204214759805.replacecommit
2024-02-04 21:48:40 2221 20240204214759805.replacecommit.inflight
2024-02-04 21:48:06 2522 20240204214759805.replacecommit.requested
2024-02-05 14:13:44 13208 20240205141148519.replacecommit
2024-02-05 14:13:21 8512 20240205141148519.replacecommit.inflight
2024-02-05 14:11:52 2522 20240205141148519.replacecommit.requested
2023-12-08 13:32:17 0 archived_$folder$
2023-12-08 13:32:25 1130 hoodie.properties
2023-12-08 13:32:21 0 metadata_$folder$
```

Also, here is the output of the command metadata validate-files ran from the hudi-cli on the corrupted table
```metadata validate-files

2024-02-02 11:28:43,123 INFO metadata.BaseTableMetadata: Listed file in partition from metadata: partition=2023/12/17/00, #files=2
%d [%thread] %-5level %logger - %msg%n%d [%thread] %-5level %logger - %msg%n????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? Partition ? File Name ? Is Present in FS ? Is Present in Metadata ? FS size ? Metadata size ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/29/00 ? 1940e6fe-febf-409b-9096-a75d01b288f7-0_0-4-1446_20231229214316193.parquet ? false ? true ? 0 ? 450079 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/29/00 ? 27107d76-bfbc-45eb-8eed-15752437efa6-0_1-5-1444_20231230214425196.parquet ? false ? true ? 0 ? 450105 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2024/01/02/00 ? 297612f5-4681-47e0-bac2-d4293ac50827-0_1-4-1414_20240102214855026.parquet ? false ? true ? 0 ? 449931 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/25/00 ? 82adb6c9-89b1-4f44-97f1-1ef235e44a4a-0_1-4-1402_20231225214603565.parquet ? false ? true ? 0 ? 449930 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/25/00 ? 3a19cf04-eca6-4e24-bd2e-75e0262d1eee-0_2-6-1406_20231226214637699.parquet ? false ? true ? 0 ? 449920 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/31/00 ? 890aa505-a02d-4adc-b4fd-62a39c28be08-0_2-4-1412_20240101214642664.parquet ? false ? true ? 0 ? 450162 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/31/00 ? e980b191-f4d0-426f-ac87-b1b773c695bf-0_2-6-1405_20231231214814370.parquet ? false ? true ? 0 ? 450084 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2024/01/01/00 ? b3a014b1-1e84-48f0-bec9-b1f3113f7d0f-0_1-4-1411_20240101214642664.parquet ? false ? true ? 0 ? 450462 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2024/01/01/00 ? 8839f2f0-8c8a-425e-b397-8151c5d2c0ae-0_0-4-1413_20240102214855026.parquet ? false ? true ? 0 ? 450512 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/26/00 ? df627807-1c95-4fa8-b253-c8317e0d79d8-0_0-4-1440_20231227220120993.parquet ? false ? true ? 0 ? 449428 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/26/00 ? 711a7fb3-b9e4-4ae8-8f07-1f162ad56453-0_0-6-1404_20231226214637699.parquet ? false ? true ? 0 ? 449332 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/21/00 ? ef8c236e-e47b-4b14-82b9-88b35a2b4b65-0_2-7-1492_20231222214842148.parquet ? false ? true ? 0 ? 450542 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/21/00 ? f0bc5273-c44a-4ff3-9b2a-c5538368d016-0_2-4-1394_20231221214633206.parquet ? false ? true ? 0 ? 450545 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/22/00 ? 274eeefe-7e73-4aae-bdd3-3d1ff25fdf96-0_0-7-1490_20231222214842148.parquet ? false ? true ? 0 ? 450676 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/22/00 ? 59f01016-e8f7-46d1-a692-7a09c540c62e-0_1-6-1468_20231223214749872.parquet ? false ? true ? 0 ? 450381 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/19/00 ? d125bb1b-e853-48e2-b24d-2324a104ac53-0_0-4-1392_20231220214948676.parquet ? false ? true ? 0 ? 450057 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/19/00 ? 3d269000-52a2-4a73-9b6b-210c8825dd43-0_0-4-1380_20231219214031893.parquet ? false ? true ? 0 ? 450073 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/20/00 ? f042aece-42c0-41c0-ac92-51d06c092be8-0_1-4-1394_20231220214948676.parquet ? false ? true ? 0 ? 450911 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/20/00 ? c23b43d5-bfab-4234-83cc-e1c757dffefc-0_1-4-1393_20231221214633206.parquet ? false ? true ? 0 ? 450961 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/27/00 ? bc2cf246-4092-49c8-81ed-eedbf84f22e4-0_1-4-1441_20231227220120993.parquet ? false ? true ? 0 ? 451679 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/27/00 ? 8a685c8a-6ff4-4964-985c-6d78d76f41a4-0_2-4-1445_20231228214506235.parquet ? false ? true ? 0 ? 451823 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/23/00 ? b57e0b51-f42a-4134-a649-1ea955f48e79-0_0-4-1407_20231224214916554.parquet ? false ? true ? 0 ? 449843 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/23/00 ? 66a52bca-6db0-4274-a20e-ecdaac67bf78-0_0-6-1467_20231223214749872.parquet ? false ? true ? 0 ? 449815 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/18/00 ? bcce485f-70ae-4a80-b36c-16e9c9596e07-0_0-4-1368_20231218215310500.parquet ? false ? true ? 0 ? 450011 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/18/00 ? 8d75a76e-ff9b-4724-b485-70170b83ca7b-0_1-4-1381_20231219214031893.parquet ? false ? true ? 0 ? 450147 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/28/00 ? 652f19b9-fa60-40f8-829d-2b540b177b6b-0_1-4-1447_20231229214316193.parquet ? false ? true ? 0 ? 450805 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/28/00 ? d40de711-9bba-4bc1-af48-f2566421c9b4-0_0-4-1443_20231228214506235.parquet ? false ? true ? 0 ? 450801 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/24/00 ? 820b6a82-b2f6-440b-9777-9dd3e2ede1d1-0_2-4-1403_20231225214603565.parquet ? false ? true ? 0 ? 449584 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/24/00 ? 2d0559ef-f850-4955-a410-16abdeda943f-0_2-4-1409_20231224214916554.parquet ? false ? true ? 0 ? 449493 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/30/00 ? cc519a61-6344-438a-9ba6-61127a11c420-0_0-5-1443_20231230214425196.parquet ? false ? true ? 0 ? 450030 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/30/00 ? 669940a4-2c3a-4d22-ac69-4efaf7b292fa-0_1-6-1406_20231231214814370.parquet ? false ? true ? 0 ? 450059 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/17/00 ? 15568146-a344-445e-b47a-266b9f15a674-0_2-4-1370_20231218215310500.parquet ? false ? true ? 0 ? 450303 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????```

**Stacktrace**
query error while reading the table :
HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://tmp-data/investments_ctr_tbl/2023/12/17/00/15568146-a344-445e-b47a-266b9f15a674-0_2-4-1370_20231218215310500.parquet (offset=0, length=450303): io.trino.hdfs.s3.TrinoS3FileSystem$UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey)

Do let us know if any other info is required regarding this issue.

Start by reproducing the issue with the listed Hudi metadata, insert_overwrite, cleaner, and S3 settings, then run the hudi-cli metadata validate-files command against the affected table. Compare the clean and replacecommit timeline entries with metadata file references; done means cleaned files are removed from metadata and Spark or Athena reads succeed with metadata listing enabled.

Written by the indexing model from the issue text.