[SUPPORT] s3 list cost increases exponentially when using COW table

We started ingesting data from kafka using spark java and wanted to understand the storage cost associated with creating a table.
Interestingly, we see that our s3 listing cost exponentially so wanted to understand if we are missing on something that can help us reduce this cost.

We do have the metadata enabled so the assumption was s3 cost will be reduced but dont see that.

When the spark job starts, we do see that the embedded timeline server starts and errors out post processing of the 1 micro batch. (Not sure if this is causing the listing cost and I believe the embedded timeline server was disabled as per this [issue](https://github.com/apache/hudi/issues/10432))

**Environment Description**

* Hudi version : 0.15.0

* Spark version : 3.4.3

* Storage (HDFS/S3/GCS..) : S3

**Stacktrace**

```
24/08/08 16:07:56 INFO TimelineService: Starting Timeline server on port :45043
24/08/08 16:07:56 INFO EmbeddedTimelineService: Started embedded timeline server at <......>
24/08/08 16:07:56 INFO BaseHoodieClient: Timeline Server already running. Not restarting the service
24/08/08 16:07:56 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20240808160601237__commit__COMPLETED__20240808160625000]}

```

```
24/08/08 16:08:02 INFO BaseHoodieClient: Embedded Timeline Server is disabled. Not starting timeline service
24/08/08 16:08:02 INFO BaseHoodieClient: Embedded Timeline Server is disabled. Not starting timeline service

```

** Hudi config **

```java
df.writeStream()
.format("hudi")
.option("hoodie.insert.shuffle.parallelism", "2")
.option("hoodie.upsert.shuffle.parallelism", "2")
.option("hoodie.delete.shuffle.parallelism", "2")
.option(EMBEDDED_TIMELINE_SERVER_ENABLE.key(), "true")

.option(HoodieWriteConfig.TBL_NAME.key(), newTName)
.option("hoodie.datasource.write.table.type", HoodieTableType.COPY_ON_WRITE.name())
.option("hoodie.datasource.write.operation", WriteOperationType.INSERT.value())
.option(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), "ts_date")
.option("checkpointLocation", newCheckPoint)

.option(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "col1,col2,col3,col4")

.option(HoodieWriteConfig.PRECOMBINE_FIELD_NAME.key(), "time_stamp")
.option(INDEX_TYPE.key(), GLOBAL_BLOOM.name())

// Clustering + Compaction config
.option(ASYNC_CLUSTERING_ENABLE.key(), "true")
.option("hoodie.clustering.plan.strategy.max.bytes.per.group", "524288000")

// Metadata Config
.option(ENABLE_METADATA_INDEX_COLUMN_STATS.key(), "true")
.option(ASYNC_INDEX_ENABLE.key(), "true")

.option(ASYNC_CLEAN.key(), "true")

.outputMode(OutputMode.Append());
```

Start with the supplied Spark streaming configuration and the embedded timeline server log lines, then compare the behavior with issue #10432. Trace which setting or component performs the S3 listings and confirm the behavior with metadata indexing enabled; done means the cause and a reproducible mitigation are documented.

Written by the indexing model from the issue text.