Executors are not scaling: they are throwing OOM when running sql cdc job for 50 million records in one-time/batch mode

### Bug Description

**What happened:**
We are trying to ingest 50M kafka records to a hudi table using delta streamer in one-time/batch mode.

**What you expected:**
We are expecting it to scale number of executors when it senses more load since we have not capped/defined maxExecutors in spark.
Following are the spark configs:
```
'spark.executor.memory=6g'
'spark.executor.cores=4'
'spark.dynamicAllocation.minExecutors=1'
'spark.executor.memoryOverhead=2g'
'spark.driver.memory=4g'
'spark.driver.cores=4'
'spark.dynamicAllocation.initialExecutors=1'
```

If we increase executor memory to 15G then it works fine but its not a ideal solution since in production, having 50M records will be a rare scenario. We are expecting hudi to scale the executors in case there are huge amount of data

**Questions:**
1. On what basis it partitions the data to be ingested?
2. Is there any option to define/control number of partitions it creates?

### Environment

**Hudi version:** 0.15.0-amzn-7
**Query engine:** (Spark/Flink/Trino etc) : Trino
**Relevant configs:** Aws EMR 7.10

### Logs and Stack Trace

_No response_

The payload names no source files or tests. Start by reproducing the DeltaStreamer one-time/batch ingestion on EMR with the listed Spark settings and inspect executor and partition behavior for the 50M-record load; done means establishing the expected scaling behavior and documenting how partitions are selected and controlled.

Written by the indexing model from the issue text.