[SUPPORT] RLI index slowing down

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**
A MongoDB table is synced by AWS DMS CDC pipeline on s3 using Glue Job.
The job execution time increases after few runs by 50%.
Table Stats:
1. Number of records at Initial Run --> 530 M
2. Avg Number of records at Incremental Runs --> 5M inserts, 20K updates, 0 deletes
3. Hudi Jars Used:
hudi-spark3.3-bundle_2.12-0.14.0.jar
hudi-aws-0.14.0.jar,
httpclient-4.5.14.jar,
spark-avro_2.12-3.5.0.jar

**To Reproduce**

Steps to reproduce the behavior:

HUDI table configuration
'hoodie.table.name': 'appsflyerevents', 'hoodie.datasource.write.precombine.field': 'upsert_ts', 'hoodie.datasource.write.recordkey.field': 'oid__id', 'hoodie.datasource.write.table.type': 'COPY_ON_WRITE', 'hoodie.datasource.hive_sync.enable': 'true', 'hoodie.datasource.hive_sync.table': 'appsflyerevents', 'hoodie.datasource.hive_sync.database': 'origin', 'hoodie.datasource.hive_sync.mode': 'hms', 'hoodie.datasource.write.hive_style_partitioning': 'true', 'hoodie.datasource.hive_sync.partition_fields': 'creation_month', 'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor', 'hoodie.datasource.write.partitionpath.field': 'creation_month', 'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.SimpleKeyGenerator', 'hoodie.datasource.write.operation': 'upsert', 'hoodie.cleaner.policy': 'KEEP_LATEST_FILE_VERSIONS', 'hoodie.cleaner.fileversions.retained': 1, 'hoodie.upsert.shuffle.parallelism': 152, 'hoodie.index.type': 'RECORD_INDEX', 'hoodie.metadata.record.index.enable': 'true', 'hoodie.metadata.record.index.growth.factor': 10, 'hoodie.metadata.record.index.max.filegroup.count': 20000, 'hoodie.metadata.record.index.min.filegroup.count': 1000, 'hoodie.metadata.record.index.max.filegroup.size': 536870912, 'hoodie.metadata.enable': 'true', 'hoodie.parquet.small.file.limit': -1, 'hoodie.metadata.clean.async': 'true', 'hoodie.metadata.keep.min.commits': '4', 'hoodie.metadata.keep.max.commits': '5', 'hoodie.datasource.meta.sync.glue.metadata_file_listing': 'true'

**Expected behavior**

The execution time should remain consistent and is not expected increase, significantly.

**Environment Description**

* Hudi version : 0.14

* Spark version : 3.3

* Hive version : NA

* Hadoop version : NA

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No

**Additional context**

Please find the spark UI attached

**Stacktrace**

```Add the stacktrace of the error.```

No source file or test is named. Start with the attached Spark UI and the listed Hudi RECORD_INDEX and metadata configuration, then compare runtime across incremental Glue runs. Done means identifying a confirmed cause or supported change that prevents execution time from increasing significantly.

Written by the indexing model from the issue text.