[SUPPORT] Slow in writing stage for upsert with Hudi 0.12.3

Hello,

Recently we migrated our datasets from Hudi 0.8 to Hudi 0.12.3 and started experiencing slowness in writing stage where parquet files are being writing to S3.

Below numbers were observed on a COW table of 12 GB in size and has 10 partitions with parquet file size roughly lying between 30MB - 300MB.

In an upsert job of 27,679 records with a total size of 26.8MB, we observed that each task in writing stage was taking up to 10 mins to write parquet file of size ranging from 30MB to 300MB. Individual task duration seems directly correlated to the size of the parquet file the task wrote, which makes sense, however, spending 10 mins on writing a 300MB parquet file into S3 seems extremely long.

Can you please help us understand what might be causing such slowness in writing stage and if there is a way to improve the performance here?

Complete spark job:

Writing stage:

Hudi commit metadata for the upsert job:

Environment Description

Hudi version : 0.12.3

Spark version : 3.1.3

Hive version : 3.1.3

Hadoop version : 3.3.3

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : no

EMR: 6.10.0/6.10.1

Additional context

Hudi configs

hoodie.metadata.enable: true
hoodie.metadata.validate: true
hoodie.cleaner.commits.retained: 72
hoodie.keep.min.commits: 100
hoodie.keep.max.commits: 150
hoodie.datasource.write.payload.class: org.apache.hudi.common.model.DefaultHoodieRecordPayload
hoodie.index.type: BLOOM
hoodie.bloom.index.parallelism: 2000
hoodie.metadata.enable: true
hoodie.datasource.write.table.type: COPY_ON_WRITE
hoodie.insert.shuffle.parallelism: 500
hoodie.datasource.write.operation: upsert
hoodie.datasource.hive_sync.partition_extractor_class: org.apache.hudi.hive.MultiPartKeysValueExtractor
hoodie.datasource.write.keygenerator.class: org.apache.hudi.keygen.ComplexKeyGenerator

No source files or tests are named. Start by reviewing the Spark write-stage metrics alongside the Hudi 0.12.3 configuration and S3/EMR environment; done means identifying the cause of the slow parquet writes and documenting an actionable performance improvement or confirmed limitation.

Written by the indexing model from the issue text.