[SUPPORT] HoodieException: Failed to apply clean commit to metadata - HUDI 0.15

Hello Community,

We are facing issue with the streaming pipeline with hudi 0.15 (hoodie.metadata.enable=True) while migrating from 0.12 to 0.15
Pipeline is running successfully with hoodie.metadata.enable=False but failing with hoodie.metadata.enable=True

Need your help to debug this issue, if you guys have faced similar issues in past please let me know

ISSUE : 1

[2024-08-11 21:22:35,835] ERROR org.apache.hudi.exception.HoodieException: Error waiting for async clean service to finish (com.pe.skull.titan.utils.SparkUtils)
[2024-08-11 21:22:35,836] ERROR writing data for pe_oms_product_oms.product_order_eligible_payment_modes_snapshot_nrt attempt: 3 (com.pe.skull.titan.utils.SparkUtils)
[2024-08-11 21:22:40,290] ERROR Service shutdown with error (org.apache.hudi.async.HoodieAsyncService)
java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Failed to apply clean commit to metadata
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) ~[?:?]

Caused by: org.apache.hudi.exception.HoodieException: Failed to apply clean commit to metadata
at org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:93) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.action.clean.CleanActionExecutor.runClean(CleanActionExecutor.java:235) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.action.clean.CleanActionExecutor.runPendingClean(CleanActionExecutor.java:200) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]

Caused by: java.lang.IllegalArgumentException: File gs://pe-skull-data/pe/processed/nrt_snapshot/data/pe_oms_product_oms/product_order_eligible_payment_modes_snapshot_nrt/.hoodie/metadata/.hoodie/20240811212153175.deltacommit.requested does not exist!
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:617) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionRequestedToInflight(HoodieActiveTimeline.java:683) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]

Caused by: java.lang.IllegalArgumentException: File gs://pe-skull-data/pe/processed/nrt_snapshot/data/pe_oms_product_oms/product_order_eligible_payment_modes_snapshot_nrt/.hoodie/metadata/.hoodie/20240811212153175.deltacommit.requested does not exist!
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:617) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]

ISSUE : 2

[2024-08-10 01:21:09,735] ERROR Failed to perform previous clean operation, instant: [==>20240810012008022__clean__INFLIGHT__20240810012011211] (org.apache.hudi.table.action.clean.CleanActionExecutor)
org.apache.hudi.exception.HoodieCommitException: Failed to complete commit 20240810012008022 due to finalize errors.
at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.finalizeWrite(BaseCommitActionExecutor.java:223) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.commit(BaseSparkCommitActionExecutor.java:303) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.autoCommit(BaseCommitActionExecutor.java:202) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.commitOnAutoCommit(BaseCommitActionExecutor.java:184) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]

Caused by: org.apache.hudi.exception.HoodieIOException: listStatus(hadoopPath: gs://pe-skull-data/pe/processed/nrt_snapshot/data/pe_mercury_vault/bk_vendor_snapshot_nrt/.hoodie/metadata/.hoodie/.temp/20240810012008022): 'gs://pe-skull-data/pe/processed/nrt_snapshot/data/pe_mercury_vault/bk_vendor_snapshot_nrt/.hoodie/metadata/.hoodie/.temp/20240810012008022' does not exist.
at org.apache.hudi.table.HoodieTable.reconcileAgainstMarkers(HoodieTable.java:784) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.HoodieTable.finalizeWrite(HoodieTable.java:696) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.HoodieSparkMergeOnReadTable.finalizeWrite(HoodieSparkMergeOnReadTable.java:230) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.finalizeWrite(BaseCommitActionExecutor.java:220) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
... 27 more
Caused by: java.io.FileNotFoundException: listStatus(hadoopPath: gs://pe-skull-data/pe/processed/nrt_snapshot/data/pe_mercury_vault/bk_vendor_snapshot_nrt/.hoodie/metadata/.hoodie/.temp/20240810012008022): 'gs://pe-skull-data/pe/processed/nrt_snapshot/data/pe_mercury_vault/bk_vendor_snapshot_nrt/.hoodie/metadata/.hoodie/.temp/20240810012008022' does not exist.
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.lambda$listStatus$11(GoogleHadoopFileSystem.java:801) ~[gcs-connector-3.0.1.jar:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547) ~[hadoop-client-api-3.3.6.jar:?]


Hoodie Options Used :

hoodie.cleaner.commits.retained: 10
hoodie.metadata.keep.max.commits: 30
hoodie.metadata.clean.async: false
hoodie.keep.max.commits: 30
hoodie.metadata.keep.min.commits: 20
hoodie.archive.async: false
hoodie.clean.automatic: true
hoodie.finalize.write.parallelism: 200
hoodie.fail.on.timeline.archiving: false
hoodie.clean.async: true
hoodie.parquet.max.file.size: 128000000
hoodie.datasource.hive_sync.support_timestamp : true
#DISABLING METADATA TO REDUCE FREQUENT CALLS TO GCS
hoodie.metadata.enable: false
hoodie.datasource.write.hive_style_partitioning : true
hoodie.parquet.small.file.limit: 100000000
hoodie.datasource.hive_sync.enable: true
hoodie.bulkinsert.shuffle.parallelism: 200
hoodie.keep.min.commits: 11
hoodie.datasource.meta.sync.enable: true
hoodie.metadata.cleaner.commits.retained: 3
hoodie.cleaner.incremental.mode: true
hoodie.commits.archival.batch: 12
hoodie.upsert.shuffle.parallelism: 200
hive_sync.support_timestamp: true
hoodie.insert.shuffle.parallelism: 200
hoodie.metadata.compact.max.delta.commits: 10
compaction.delta_commits: 5
metadata.compaction.delta_commits: 10
hoodie.compact.inline.max.delta.commits: 5
hoodie.archive.automatic: true
hoodie.cleaner.parallelism: 200

Start with CleanActionExecutor.runPendingClean, BaseActionExecutor.writeTableMetadata, HoodieTable.reconcileAgainstMarkers, and HoodieActiveTimeline.transitionState. Reproduce the Hudi 0.15 streaming failure on GCS with metadata enabled, then determine why the requested deltacommit and .temp paths are missing; done requires a confirmed cause and documented remediation.

Written by the indexing model from the issue text.