Hudi Record Index Initilization: Poor performance and job failures during Bulk Inserting into metadata table

### Bug Description

**What happened:**
When attempting to switch from GLOBAL_BLOOM index to RECORD_INDEX for the first time the write job fails when initializing the record index metadata.

The job consistantly fails during this operation:
`Bulk inserting at 20260818190857353010 into metadata table transactions_all_metadata keyBy at `

The table size of the table being written to is ~2TB
There is some data skew in the table: some partitions have more data than other partitions.

We ran this exact same hudi write job with RECORD_INDEX on a table which is ~500GB and it succeeded. So it appears hudi is failing to scale its record index metadata generation on larger tables.

**What you expected:**
Hudi would generate the record_index metadata and perform the write job without the job crashing.

**Steps to reproduce:**
1. Create a hudi table of approximately 2 TB using the configurations listed below. But change RECORD_INDEX to GLOBAL_BLOOM.
2. Create an emr serverless spark job which perform a hudi write with the following configurations listed below. The input data size is roughly a few GB
3. Wait for the job to crash.

### Environment

**Hudi version:**
`0.14.0-amzn-1`
**Query engine:** (Spark/Flink/Trino etc)
`Spark 3.5.0`
**Relevant configs:**
Hudi Configuration Options:
```hoodie.bootstrap.parallelism = 1920
hoodie.bulkinsert.shuffle.parallelism = 1920
hoodie.clean.automatic = false
hoodie.cleaner.policy.failed.writes = LAZY
hoodie.compact.inline = false
hoodie.datasource.hive_sync.partition_fields = warehouse,year,month
hoodie.datasource.write.hive_style_partitioning = true
hoodie.datasource.write.keygenerator.class = org.apache.hudi.keygen.ComplexKeyGenerator
hoodie.datasource.write.operation = upsert
hoodie.datasource.write.partitionpath.field = warehouse,year,month
hoodie.datasource.write.payload.class = org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
hoodie.datasource.write.precombine.field = CaptureDate
hoodie.datasource.write.reconcile.schema = true
hoodie.datasource.write.recordkey.field = uuid
hoodie.datasource.write.table.type = MERGE_ON_READ
hoodie.delete.shuffle.parallelism = 1920
hoodie.filesystem.operation.retry.enable = true
hoodie.filesystem.operation.retry.max_interval_ms = 5000
hoodie.filesystem.operation.retry.max_numbers = 10
hoodie.finalize.write.parallelism = 1920
hoodie.global.index.reconcile.parallelism = 1920
hoodie.global.simple.index.parallelism = 1920
hoodie.index.type = RECORD_INDEX
hoodie.insert.shuffle.parallelism = 1920
hoodie.keep.max.commits = 210
hoodie.keep.min.commits = 200
hoodie.markers.delete.parallelism = 1920
hoodie.meta.sync.metadata_file_listing = true
hoodie.metadata.enable = true
hoodie.metadata.record.index.enable = true
hoodie.parquet.max.file.size = 125829120
hoodie.parquet.small.file.limit = 100663296
hoodie.record.index.update.partition.path = true
hoodie.rollback.parallelism = 1920
hoodie.schema.on.read.enable = false
hoodie.simple.index.parallelism = 1920
hoodie.table.name = transactions_all
hoodie.table.services.enabled = true
hoodie.upsert.shuffle.parallelism = 1920
hoodie.write.concurrency.mode = OPTIMISTIC_CONCURRENCY_CONTROL
hoodie.write.lock.dynamodb.billing_mode = PAY_PER_REQUEST
hoodie.write.lock.dynamodb.endpoint_url = dynamodb.us-east-1.amazonaws.com
hoodie.write.lock.dynamodb.partition_key = ******
hoodie.write.lock.dynamodb.region = us-east-1
hoodie.write.lock.dynamodb.table = ******
hoodie.write.lock.provider = org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
```
EMR Serverless Configuration:
```
{
"applicationConfiguration": [
{
"classification": "emrfs-site",
"configurations": [],
"properties": {
"fs.s3.aimd.maxAttempts": "1000",
"fs.s3.aimd.enabled": "true",
"fs.s3.maxRetries": "50"
}
},
{
"classification": "hive-site",
"configurations": [],
"properties": {
"hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
}
},
{
"classification": "spark-defaults",
"configurations": [],
"properties": {
"spark.executor.memory": "50G",
"spark.driver.memory": "50G",
"spark.emr-serverless.driver.disk": "200G",
"spark.driver.maxResultSize": "20G",
"spark.driver.cores": "8",
"spark.emr-serverless.memoryOverheadFactor": "0.2",
"spark.executor.cores": "8",
"spark.hadoop.fs.s3.serverSideEncryption.kms.keyId": "******",
"spark.hadoop.fs.s3.enableServerSideEncryption": "true",
"spark.dynamicAllocation.maxExecutors": "200",
"spark.hadoop.fs.s3.maxConnections": "9999",
"spark.emr-serverless.executor.disk": "200G",
"spark.executorEnv.vfnEmrVersion": "emr-7.0.0",
"spark.executorEnv.vfnEnvironment": "prod",
"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
"spark.emr-serverless.driverEnv.vfnEnvironment": "prod",
"spark.jars": "/usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/hudi/hudi-aws-bundle.jar",
"spark.emr-serverless.driverEnv.vfnEmrVersion": "emr-7.0.0"
}
}
],
"monitoringConfiguration": {
"managedPersistenceMonitoringConfiguration": {
"enabled": true,
"encryptionKeyArn": {}
},
"s3MonitoringConfiguration": {
"encryptionKeyArn": "******",
"logUri": "****&"
},
"cloudWatchLoggingConfiguration": {
"enabled": true,
"logGroupName": "*****",
"logStreamNamePrefix": {},
"encryptionKeyArn": {},
"logTypes": {
"SPARK_DRIVER": [
"STDOUT",
"STDERR"
],
"SPARK_EXECUTOR": [
"STDOUT",
"STDERR"
]
}
},
"prometheusMonitoringConfiguration": {}
},
"diskEncryptionConfiguration": {}
}
```

Spark Properties:
```
spark.app.id | 00g83gdn47a8r80b
spark.app.initial.jar.urls | s3://***********,spark://[2600:1f10:4ace:6700:ad82:9311:5abc:965a]:38207/jars/hudi-aws-bundle-0.14.0-amzn-1.jar,spark://[2600:1f10:4ace:6700:ad82:9311:5abc:965a]:38207/jars/hudi-spark3.5-bundle_2.12-0.14.0-amzn-1.jar
spark.app.name | ******
spark.app.startTime | 1787083716722
spark.app.submitTime | 1787083716413
spark.authenticate | true
spark.blacklist.decommissioning.enabled | true
spark.blacklist.decommissioning.timeout | 1h
spark.decommissioning.timeout.threshold | 20
spark.default.parallelism | 1920
spark.driver.bindAddress | [fd00::2%eth0]
spark.driver.cores | 4
spark.driver.defaultJavaOptions | -XX:OnOutOfMemoryError='kill -9 %p'
spark.driver.extraClassPath | /usr/lib/livy/rsc-jars/*:/usr/lib/livy/repl_2.12-jars/*:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/goodies/lib/emr-serverless-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/docker/usr/lib/hadoop-lzo/lib/*:/docker/usr/lib/hadoop/hadoop-aws.jar:/docker/usr/share/aws/aws-java-sdk/*:/docker/usr/share/aws/emr/emrfs/conf:/docker/usr/share/aws/emr/emrfs/lib/*:/docker/usr/share/aws/emr/emrfs/auxlib/*:/docker/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/docker/usr/share/aws/emr/security/conf:/docker/usr/share/aws/emr/security/lib/*:/docker/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/docker/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/docker/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/docker/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/usr/share/aws/redshift/jdbc/RedshiftJDBC.jar:/usr/share/aws/redshift/spark-redshift/lib/*:/usr/share/aws/iceberg/lib/iceberg-emr-common.jar:/usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar:/usr/lib/hudi/hudi-spark-bundle.jar:/usr/lib/hudi/hudi-aws-bundle.jar
spark.driver.extraJavaOptions | -Djava.net.preferIPv6Addresses=false -XX:OnOutOfMemoryError='kill -9 %p' -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -XX:OnOutOfMemoryError='kill -9 %p'
spark.driver.extraLibraryPath | /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native
spark.driver.host | [2600:1f10:4ace:6700:ad82:9311:5abc:965a]
spark.driver.maxResultSize | 5G
spark.driver.memory | 26G
spark.driver.port | 38207
spark.dynamicAllocation.enabled | true
spark.dynamicAllocation.initialExecutors | 3
spark.dynamicAllocation.maxExecutors | 62
spark.dynamicAllocation.minExecutors | 0
spark.dynamicAllocation.shuffleTracking.enabled | true
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout | 1s
spark.emr-serverless.allocation.executor.timeout | 300s
spark.emr-serverless.client.create.batch.size | 100
spark.emr-serverless.client.describe.batch.size | 100
spark.emr-serverless.client.release.batch.size | 100
spark.emr-serverless.driver.disk | 200G
spark.emr-serverless.driverEnv.SPARK_LOCAL_IP | [fd00::2%eth0]
spark.emr-serverless.executor.disk | 200G
spark.emr-serverless.fluentd.eventLog.custom.chunking.enabled | false
spark.emr-serverless.initialExecutorTimeout | 1200000
spark.emr-serverless.lakeformation.enabled |  
spark.emr-serverless.maxPendingExecutors | 1000
spark.emr-serverless.memoryOverheadFactor | 0.1
spark.eventLog.dir | file:///var/log/spark/apps
spark.eventLog.enabled | true
spark.eventLog.rotation.enabled | true
spark.eventLog.rotation.interval | 300s
spark.eventLog.rotation.maxFilesToRetain | 2
spark.eventLog.rotation.minFileSize | 1m
spark.executor.cores | 4
spark.executor.defaultJavaOptions | -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseParallelGC -XX:InitiatingHeapOccupancyPercent=70 -XX:OnOutOfMemoryError='kill -9 %p'
spark.executor.extraClassPath | /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/goodies/lib/emr-serverless-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/docker/usr/lib/hadoop-lzo/lib/*:/docker/usr/lib/hadoop/hadoop-aws.jar:/docker/usr/share/aws/aws-java-sdk/*:/docker/usr/share/aws/emr/emrfs/conf:/docker/usr/share/aws/emr/emrfs/lib/*:/docker/usr/share/aws/emr/emrfs/auxlib/*:/docker/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/docker/usr/share/aws/emr/security/conf:/docker/usr/share/aws/emr/security/lib/*:/docker/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/docker/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/docker/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/docker/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/usr/share/aws/redshift/jdbc/RedshiftJDBC.jar:/usr/share/aws/redshift/spark-redshift/lib/*:/usr/share/aws/iceberg/lib/iceberg-emr-common.jar:/usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar:/usr/lib/hudi/hudi-spark-bundle.jar:/usr/lib/hudi/hudi-aws-bundle.jar
spark.executor.extraJavaOptions | -Djava.net.preferIPv6Addresses=false -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseParallelGC -XX:InitiatingHeapOccupancyPercent=70 -XX:OnOutOfMemoryError='kill -9 %p' -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseParallelGC -XX:InitiatingHeapOccupancyPercent=70 -XX:OnOutOfMemoryError='kill -9 %p'
spark.executor.extraLibraryPath | /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native
spark.executor.id | driver
spark.executor.instances | 3
spark.executor.memory | 26G
spark.executorEnv.vfnEmrVersion | emr-7.0.0
spark.executorEnv.vfnEnvironment | prod
spark.files.fetchFailure.unRegisterOutputOnHost | true
spark.hadoop.aws.region | us-east-1
spark.hadoop.dynamodb.region | us-east-1
spark.hadoop.fs.defaultFS | file:///
spark.hadoop.fs.s3.customAWSCredentialsProvider | com.amazonaws.auth.DefaultAWSCredentialsProviderChain
spark.hadoop.fs.s3.enableServerSideEncryption | true
spark.hadoop.fs.s3.getObject.initialSocketTimeoutMilliseconds | 2000
spark.hadoop.fs.s3.maxConnections | 9999
spark.hadoop.fs.s3.serverSideEncryption.kms.keyId | *****
spark.hadoop.fs.s3a.aws.credentials.provider | software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider
spark.hadoop.fs.s3a.committer.magic.enabled | true
spark.hadoop.fs.s3a.committer.name | magicv2
spark.hadoop.hive.metastore.client.factory.class | com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version.emr_internal_use_only.EmrFileSystem | 2
spark.hadoop.mapreduce.fileoutputcommitter.cleanup-failures.ignored.emr_internal_use_only.EmrFileSystem | true
spark.hadoop.mapreduce.output.fs.optimized.committer.enabled | true
spark.hadoop.parquet.crypto.factory.class | org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory
spark.hadoop.parquet.encryption.kms.client.class | *******
spark.history.fs.logDirectory | file:///var/log/spark/apps
spark.history.ui.port | 18080
spark.jars | file:/usr/lib/hudi/hudi-spark3.5-bundle_2.12-0.14.0-amzn-1.jar,file:/usr/lib/hudi/hudi-aws-bundle-0.14.0-amzn-1.jar,s3://*******
spark.kryoserializer.buffer.max.mb | 256
spark.master | custom:emr-serverless
spark.repl.local.jars | file:///usr/lib/hudi/hudi-spark3.5-bundle_2.12-0.14.0-amzn-1.jar,file:///usr/lib/hudi/hudi-aws-bundle-0.14.0-amzn-1.jar
spark.resourceManager.cleanupExpiredHost | true
spark.scheduler.mode | FIFO
spark.serializer | org.apache.spark.serializer.KryoSerializer
spark.sql.adaptive.enabled | true
spark.sql.catalogImplementation | hive
spark.sql.emr.internal.extensions | com.amazonaws.emr.spark.EmrSparkSessionExtensions
spark.sql.hive.metastore.sharedPrefixes | software.amazon.awssdk.services.dynamodb
spark.sql.legacy.avro.datetimeRebaseModeInRead | LEGACY
spark.sql.legacy.avro.datetimeRebaseModeInWrite | LEGACY
spark.sql.legacy.parquet.datetimeRebaseModeInRead | LEGACY
spark.sql.legacy.parquet.datetimeRebaseModeInWrite | LEGACY
spark.sql.legacy.parquet.int96RebaseModeInRead | LEGACY
spark.sql.legacy.parquet.int96RebaseModeInWrite | LEGACY
spark.sql.parquet.fs.optimized.committer.optimization-enabled | true
spark.sql.parquet.output.committer.class | com.amazon.emr.committer.EmrOptimizedSparkSqlParquetOutputCommitter
spark.sql.shuffle.partitions | 1920
spark.ssl.internode.enabled | false
spark.ssl.ui.enabled | false
spark.stage.attempt.ignoreOnDecommissionFetchFailure | true
spark.submit.customResourceManager.submit.class | org.apache.spark.deploy.emrserverless.submit.EmrServerlessClientApplication
spark.submit.deployMode | client
spark.submit.pyFiles |  
spark.ui.custom.executor.log.url | /logs/{{CONTAINER_ID}}/{{FILE_NAME}}.gz
spark.ui.enabled | true
spark.ui.killEnabled | false
spark.ui.port | 4040
spark.yarn.heterogeneousExecutors.enabled | false

```

### Logs and Stack Trace

The job fails at this stage consistently:

Stage Id ▾ | Description | Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write | Failure Reason
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
16 | Bulk inserting at 20260818190857353010 into metadata table transactions_all_metadatakeyBy at SparkHoodieMetadataBulkInsertPartitioner.java:74+detailsorg.apache.spark.api.java.AbstractJavaRDDLike.keyBy(JavaRDDLike.scala:45) org.apache.hudi.metadata.SparkHoodieMetadataBulkInsertPartitioner.repartitionRecords(SparkHoodieMetadataBulkInsertPartitioner.java:74) org.apache.hudi.metadata.SparkHoodieMetadataBulkInsertPartitioner.repartitionRecords(SparkHoodieMetadataBulkInsertPartitioner.java:40) org.apache.hudi.table.action.commit.SparkBulkInsertHelper.bulkInsert(SparkBulkInsertHelper.java:126) org.apache.hudi.table.action.commit.SparkBulkInsertHelper.bulkInsert(SparkBulkInsertHelper.java:81) org.apache.hudi.table.action.deltacommit.SparkBulkInsertPreppedDeltaCommitActionExecutor.execute(SparkBulkInsertPreppedDeltaCommitActionExecutor.java:52) org.apache.hudi.table.HoodieSparkMergeOnReadTable.bulkInsertPrepped(HoodieSparkMergeOnReadTable.java:139) org.apache.hudi.table.HoodieSparkMergeOnReadTable.bulkInsertPrepped(HoodieSparkMergeOnReadTable.java:88) org.apache.hudi.client.SparkRDDWriteClient.bulkInsertPreppedRecords(SparkRDDWriteClient.java:237) org.apache.hudi.client.SparkRDDWriteClient.bulkInsertPreppedRecords(SparkRDDWriteClient.java:63) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.commitInternal(HoodieBackedTableMetadataWriter.java:1129) org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.bulkCommit(SparkHoodieBackedTableMetadataWriter.java:130) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFromFilesystem(HoodieBackedTableMetadataWriter.java:445) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeIfNeeded(HoodieBackedTableMetadataWriter.java:278) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.(HoodieBackedTableMetadataWriter.java:182) org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.(SparkHoodieBackedTableMetadataWriter.java:95) org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.create(SparkHoodieBackedTableMetadataWriter.java:72) org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:287) org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:273) org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1263) | 2026/08/18 20:16:20 | 29 min | 25518/6573 (24760 failed) (65 killed: Stage cancelled: Job aborted due to stage failure: Task 891 in stage 16.0 failed 4 times, most recent failure: Lost task 891.10 in stage 16.0 (TID 41417) ([2600:1f10:4ace:6700:201d:72db:f403:b6fd] executor 371): ExecutorLostFailure (executor 371 exited caused by one of the running tasks) Reason: Unknown executor exit code (137) (died from signal 9?) Driver stacktrace:) | 154.5 GiB |   |   | 561.0 GiB | Job aborted due to stage failure: Task 891 in stage 16.0 failed 4 times, most recent failure: Lost task 891.10 in stage 16.0 (TID 41417) ([2600:1f10:4ace:6700:201d:72db:f403:b6fd] executor 371): ExecutorLostFailure (executor 371 exited caused by one of the running tasks) Reason: Unknown executor exit code (137) (died from signal 9?)+detailsJob aborted due to stage failure: Task 891 in stage 16.0 failed 4 times, most recent failure: Lost task 891.10 in stage 16.0 (TID 41417) ([2600:1f10:4ace:6700:201d:72db:f403:b6fd] executor 371): ExecutorLostFailure (executor 371 exited caused by one of the running tasks) Reason: Unknown executor exit code (137) (died from signal 9?) Driver stacktrace:

Before the failure above, the job appears to be creating 677 record index file groups:

Stage Id ▾ | Description | Submitted | Duration | Tasks: Succeeded/Total | Input | Output | Shuffle Read | Shuffle Write
-- | -- | -- | -- | -- | -- | -- | -- | --
15 | Creating 677 file groups for partition record_index with base fileId record-index- at instant time 20260818190857353010foreach at HoodieSparkEngineContext.java:155+detailsorg.apache.spark.api.java.AbstractJavaRDDLike.foreach(JavaRDDLike.scala:45) org.apache.hudi.client.common.HoodieSparkEngineContext.foreach(HoodieSparkEngineContext.java:155) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFileGroups(HoodieBackedTableMetadataWriter.java:748) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFromFilesystem(HoodieBackedTableMetadataWriter.java:441) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeIfNeeded(HoodieBackedTableMetadataWriter.java:278) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.(HoodieBackedTableMetadataWriter.java:182) org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.(SparkHoodieBackedTableMetadataWriter.java:95) org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.create(SparkHoodieBackedTableMetadataWriter.java:72) org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:287) org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:273) org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1263) org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1303) org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:139) org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:224) org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:431) org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132) org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150) org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73) | 2026/08/18 20:16:17 | 1 s | 677/677 |   |   |   |  
14 | Record Index: reading record keys from 6573 base filescount at HoodieJavaRDD.java:115+detailsorg.apache.spark.api.java.AbstractJavaRDDLike.count(JavaRDDLike.scala:45) org.apache.hudi.data.HoodieJavaRDD.count(HoodieJavaRDD.java:115) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFromFilesystem(HoodieBackedTableMetadataWriter.java:435) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeIfNeeded(HoodieBackedTableMetadataWriter.java:278) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.(HoodieBackedTableMetadataWriter.java:182) org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.(SparkHoodieBackedTableMetadataWriter.java:95) org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.create(SparkHoodieBackedTableMetadataWriter.java:72) org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:287) org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:273) org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1263) org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1303) org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:139) org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:224) org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:431) org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132) org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150) org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73) org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84) | 2026/08/18 20:13:43 | 2.6 min | 6573/6573 | 2.0 TiB |   |   |  
13 | Record Index: reading record keys from 6573 base filescount at HoodieJavaRDD.java:115+detailsorg.apache.spark.api.java.AbstractJavaRDDLike.count(JavaRDDLike.scala:45) org.apache.hudi.data.HoodieJavaRDD.count(HoodieJavaRDD.java:115) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeRecordIndexPartition(HoodieBackedTableMetadataWriter.java:517) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeFromFilesystem(HoodieBackedTableMetadataWriter.java:420) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeIfNeeded(HoodieBackedTableMetadataWriter.java:278) org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.(HoodieBackedTableMetadataWriter.java:182) org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.(SparkHoodieBackedTableMetadataWriter.java:95) org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.create(SparkHoodieBackedTableMetadataWriter.java:72) org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:287) org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:273) org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1263) org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1303) org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:139) org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:224) org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:431) org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132) org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150) org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75) org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73) | 2026/08/18 20:09:11 | 4.5 min | 6573/6573 |   |   |  

Start with record-index metadata initialization and the “Bulk inserting” operation into the metadata table, using the supplied RECORD_INDEX, Spark, and EMR Serverless configuration to reproduce the failure. Done means a roughly 2 TB table with skewed partitions completes record-index generation and the write job without crashing.

Written by the indexing model from the issue text.