[SUPPORT]Checkpoint Fails with Timeout while waiting for instant initialize When Migrating Data from Kudu to Hudi COW Bucketed Table

Problem Description
We encountered the following issues while using Apache Hudi for data migration and real-time writing:

Scenario 1:

Migrating data from Kudu to a Hudi MOR bucketed table, then writing data from MySQL via Kafka to the Hudi MOR bucketed table works fine.

Scenario 2:

Migrating data from Kudu to a Hudi COW bucketed table, then writing data from MySQL via Kafka to the Hudi COW bucketed table fails to generate commits, and the checkpoint fails.

Error Log
Here is the error log when the task fails:

org.apache.flink.streaming.runtime.tasks.AsynccheckpointRunnable []. bucket_write default_databases.table_cow ardemmewlcos part of checkpoint 1 could not be completed.

java.util.concurrent.cancellationException:null

org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 2 for operator bucket_write: default_database.table_con (3/4). failure reason: Checkpoint was declined.

Caused by :org.apache.hudi.exception.HoodieException:Timeout(1201000ms) while waiting for instant initialize

switched from RUNNING to FAILED with failure cause: jave.io.I0Exception: could not perforn checkpoint 2 for operator bucket_write: default_database.table_cow(3/4)#0.

Configuration Parameters
Here is our Hudi table configuration:

options.put("hoodie.upsert.shuffle.parallelism", "20");
options.put("hoodie.insert.shuffle.parallelism", "20");
options.put("write.operation", "upsert");
options.put(FlinkOptions.TABLE_TYPE.key(), name);
options.put(FlinkOptions.PRECOMBINE_FIELD.key(),precombing);
options.put(FlinkOptions.PRE_COMBINE.key(), "true");
options.put("hoodie.clean.automatic", "true");
options.put("hoodie.cleaner.policy", "KEEP_LATEST_COMMITS");
options.put("hoodie.cleaner.commits.retained", "5");
options.put("hoodie.clean.async", "true");
options.put("hoodie.archive.min.commits", "20");
options.put("hoodie.archive.max.commits", "30");
options.put("hoodie.clean.parallelism", "20");
options.put("hoodie.archive.parallelism", "20");
options.put("hoodie.write.concurrency.mode","optimistic_concurrency_control");
options.put("write.tasks", "20");
options.put("index.type","BUCKET");
options.put("hoodie.bucket.index.num.buckets","80");
options.put("hoodie.index.bucket.engine","SIMPLE");

Checkpoint Configuration
We tested various checkpoint timeout and interval configurations, but the issue persists:

env.getCheckpointConfig().setCheckpointTimeout(5*60*1000L);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(60*1000L);

Steps to Reproduce
Migrate data from Kudu to a Hudi COW bucketed table.

Write data from MySQL via Kafka to the Hudi COW bucketed table.

The task fails with the error Timeout while waiting for instant initialize.

Expected Behavior
The task should generate commits normally, and the checkpoint should succeed.

Actual Behavior
The task fails, no commits are generated, and the checkpoint fails with an error.

Hudi version : 0.14.0

Spark version : 2.4.7

Hive version : 3.1.3

Hadoop version : 3.1.1
Further Questions
Checkpoint Timeout Issue:
The error log mentions Timeout while waiting for instant initialize. Is this related to the initialization mechanism of the Hudi COW table? Are there ways to optimize the initialization time?

COW Table Write Performance:
Is the write performance of COW tables slower than MOR tables? Are there optimization suggestions for COW tables?

Impact of Bucketed Table:
Does the bucketed table have a specific impact on the write performance of COW tables? Are there optimization configurations for bucketed tables?

Checkpoint Configuration:
We tried various checkpoint timeout and interval configurations, but the issue persists. Are there recommended checkpoint configurations?

Summary
We would like to know:

Why does the Hudi COW bucketed table encounter checkpoint timeout issues during writes?

Are there optimization suggestions for COW table write performance?

Does the bucketed table have a specific impact on COW table write performance?

Are there recommended checkpoint configurations?

Thank you for your help!

No source files, tests, or entry points are named. Start with the listed reproduction steps and the Flink checkpoint error, then compare the COW and MOR bucketed-table configurations and the checkpoint settings. Done means identifying the cause of the instant-initialize timeout and documenting or implementing a verified fix that allows commits and checkpoints to succeed.

Written by the indexing model from the issue text.