[SUPPORT] Excessive S3 API calls to hudi rollback files on upsert job

**Describe the problem you faced**

We are experiencing excessive S3 API calls and poor performance in our Hudi-based datalake pipeline cdc application. Each spark executor in the application is opening the 130 rollback files in S3 repeatedly, approximately 1000 times per rollback file per executor. This is causing S3 503 throttling errors and significant performance degradation during our CDC data processing pipeline.

**To Reproduce**

Steps to reproduce the behavior:

1. Set up a hudi table in s3 with approximately 130 rollback files
1. Run an EMR Serverlesss Spark application that upserts data into this table using the configurations provided below
5. Observe S3 access patterns throgh Spark executor stderr.gz logs in Spark History Server

**Expected behavior**

Rollback files data should be read efficiently with minimal redundant S3 API calls. Each executor should not need to repeatedly list the same rollback files 1000+ times, and the application should not encounter S3 throttling errors due to excessive s3 api operations. Rollback file data should likely be cached after being read for the first time.

**Environment Description**

* Hudi version : 0.13.1-amzn-0

* Spark version : 3.4.0-amzn-0

* Hive version : 3.1.3-amzn-5

* Hadoop version : 3.3.3-amzn-4

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No (AWS EMR Serverless)

Hoodie configuration:

```
"hoodie.datasource.write.recordkey.field": "uuid",
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
"hoodie.datasource.write.payload.class": "org.apache.hudi.common.modelOverwriteWithLatestAvroPayload",
"hoodie.table.name": "*********",
"hoodie.datasource.write.table.type": "merge_on_read",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.schema.on.read.enable": "false",
"hoodie.datasource.write.reconcile.schema": "true",
"hoodie.datasource.write.precombine.field": "CaptureDate",
"hoodie.compact.inline": "true",
"hoodie.simple.index.parallelism": "1920",
"hoodie.global.simple.index.parallelism": "1920",
"hoodie.index.type": "global_bloom",
"hoodie.index.class": "org.apache.hudi.index.bloom.HoodieSemiGlobalWarehouseBloomIndex",
"hoodie.datasource.write.partitionpath.field": "****,****,****",
"hoodie.datasource.hive_sync.partition_fields": "****,****,****",
"hoodie.upsert.shuffle.parallelism": "1920",
"hoodie.datasource.meta_sync.condition.sync": "true",
"hoodie.datasource.meta.sync.enable": "true",
"hoodie.datasource.hive_sync.enable": "true",
"hoodie.datasource.hive_sync.mode": "hms",
"hoodie.datasource.hive_sync.use_jdbc": "false",
"hoodie.datasource.hive_sync.skip_ro_suffix": "true",
"hoodie.datasource.hive_sync.database": "p*******4",
"hoodie.datasource.hive_sync.table": "*******",
"hoodie.datasource.hive_sync.support_timestamp": "true",
"hoodie.meta.sync.client.tool.class": "org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool",
"hoodie.metadata.enable": "true",
"hoodie.meta.sync.metadata_file_listing": "true",
"hoodie.archive.automatic": "true",
"hoodie.keep.min.commits": "200",
"hoodie.keep.max.commits": "210",
"hoodie.clean.automatic": "true",
"hoodie.cleaner.parallelism": "1920",
"hoodie.cleaner.policy": "KEEP_LATEST_COMMITS",
"hoodie.cleaner.commits.retained": "90",
"hoodie.parquet.max.file.size": "1073741824",
"hoodie.parquet.small.file.limit": "838860800",
"hoodie.write.concurrency.mode": "OPTIMISTIC_CONCURRENCY_CONTROL",
"hoodie.cleaner.policy.failed.writes": "LAZY",
"hoodie.write.lock.provider": "org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider",
"hoodie.write.lock.dynamodb.table": "p*******r",
"hoodie.write.lock.dynamodb.partition_key": "*******",
"hoodie.write.lock.dynamodb.region": "us-east-1",
"hoodie.write.lock.dynamodb.endpoint_url": "dynamodb.us-east-1.amazonaws.com",
"hoodie.write.lock.dynamodb.billing_mode": "PAY_PER_REQUEST"
```

AWS EMR Serverless

- EMR Release Label: emr-6.12.0

- Application Limits
Maximum CPU (vCPU): 2250
Maximum memory (GB): 17550
Maximum disk (GB): 112500

- Job Configurations
```
fs.s3.aimd.maxAttempts: 1000
fs.s3.maxRetries: 50
fs.s3.aimd.enabled: true
hive.metastore.client.factory.class: com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
spark.executor.memory: 26G
spark.driver.memory: 26G
spark.emr-serverless.driver.disk: 200G
spark.driver.maxResultSize: 5G
spark.driver.cores: 4
spark.emr-serverless.memoryOverheadFactor: 0.1
spark.hadoop.fs.s3.serverSideEncryption.kms.keyId: alias/********
spark.hadoop.fs.s3.enableServerSideEncryption: true
spark.hadoop.fs.s3.maxConnections: 9999
spark.emr-serverless.executor.disk: 200G
spark.hadoop.hive.metastore.client.factory.class: com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
spark.serializer: org.apache.spark.serializer.KryoSerializer
spark.jars: /usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/hudi/hudi-aws-bundle.jar
```

- Java Runtime: Java 17

**Additional context**

Hudi Table Size:
- The hudi table which we are upserting into has an estimated size of 10 million files and 20 TB's of data

Why Hudi Verion 0.13.1 is Required:
- The company I work for is strictly unable to change hudi versions for various reasons. Updating to a later version of Hudi would update our table version and break the reader applications that consume this data. We don't control these reader applications and cannot risk breaking any of them by requiring updates they may not implement. Changing our version of Hudi is therefore not a possible solution for us.

Root Cause of Rollback Accumulation:
- The 130+ rollback files have accumulated due to occasional job failures over the past year. Our archive retention settings (hoodie.keep.min.commits: 200, hoodie.keep.max.commits: 210) are intentionally configured to maintain extensive commit history for downstream processing requirements. These settings are necessary for our specific use case requirements where we need extensive commit history for downstream processes. Additionally, we maintain this large retention window to ensure the archiver never removes timeline entries before the cleaner processes the corresponding commits, preventing potential data consistency issues.

**Stacktrace**

Spark history server executor stderr.gz
```
25/09/09 15:03:35 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240807003441694.rollback' for reading
25/09/09 15:03:35 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240807003441694.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240812203551496.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240812203551496.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240807003441694.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822014339510.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240807003441694.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822014339510.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822015759845.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822015759845.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240812203551496.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240812203551496.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822014339510.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822014339510.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822015759845.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822015759845.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822182728413.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240823030824469.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170803546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822182728413.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170807059.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240823030824469.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170810335.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170803546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170813517.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822182728413.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240822182728413.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170807059.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170817281.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240823030824469.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240823030824469.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170820570.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170810335.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170803546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240828135048938.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170813517.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170803546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170817281.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170807059.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170807059.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240830165641261.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170820570.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170810335.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150914282.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170810335.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240828135048938.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170813517.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170813517.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240830165641261.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150925674.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170817281.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170817281.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150930546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150914282.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170820570.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240827170820570.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150934788.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150925674.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240828135048938.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240828135048938.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150930546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240830165641261.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240830165641261.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150934788.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150914282.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150914282.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906151203832.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150925674.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150925674.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150930546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906151203832.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150930546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150934788.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906150934788.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906151203832.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240906151203832.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240909140753262.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240909140753262.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240909184625794.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240910130154536.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240909140753262.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240909184625794.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240909140753262.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240910130154536.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240920171740272.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240909184625794.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240921153540839.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240909184625794.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240910130154536.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240920171740272.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240922182329367.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240910130154536.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240923121259866.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240921153540839.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240920171740272.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240923173606554.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240922182329367.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240920171740272.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241005054221114.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240921153540839.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240923121259866.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240921153540839.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241006133732033.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240922182329367.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240923173606554.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241210190944069.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240922182329367.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240923121259866.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241005054221114.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240923121259866.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240923173606554.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241213131144553.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241005054221114.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241006133732033.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20240923173606554.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241217164635772.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241005054221114.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241210190944069.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241217164639571.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241006133732033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241218180918757.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241213131144553.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241006133732033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241210190944069.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241218180923701.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241217164635772.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241210190944069.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203510872.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241217164639571.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241213131144553.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241213131144553.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203512529.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241218180918757.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241217164635772.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203514074.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241217164635772.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241218180923701.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241217164639571.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250222083508448.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241217164639571.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203510872.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241218180918757.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241218180918757.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203512529.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241218180923701.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250306172823755.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203514074.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20241218180923701.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203510872.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250314180840033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250222083508448.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203510872.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203512529.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203512529.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250507004949483.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203514074.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250306172823755.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250220203514074.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250222083508448.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250314180840033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250522083838588.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250222083508448.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250306172823755.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250507004949483.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250528103012780.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250306172823755.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214319692.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250314180840033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250522083838588.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250314180840033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214342091.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250507004949483.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214356578.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250528103012780.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250507004949483.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214423608.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250522083838588.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214319692.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250522083838588.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214342091.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214443938.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250528103012780.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214356578.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214501281.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250528103012780.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214319692.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214521140.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214319692.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214423608.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214342091.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214342091.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214538229.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214443938.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214356578.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214356578.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214548343.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214423608.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214501281.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214423608.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123207319.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214521140.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214443938.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214443938.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123302138.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214538229.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214501281.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214501281.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123339708.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214548343.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214521140.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214521140.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609234916106.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123207319.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214538229.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250611192638465.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123302138.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214548343.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214538229.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123339708.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123207319.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250611192815332.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250608214548343.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250611192821763.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609234916106.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123302138.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123207319.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250611192827451.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250611192638465.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123339708.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening 's3://********/******.all_hudi/.hoodie/20250609123302138.rollback' for reading
... continues this pattern consistently until 25/09/09 17:17:54, does so on multiple spark executors
```

Start with the Spark executor stderr.gz logs in Spark History Server and the rollback-file openings described in the report. Reproduce the upsert against an S3 Hudi table with accumulated rollback files, then verify that repeated reads are reduced and S3 throttling no longer occurs.

Written by the indexing model from the issue text.