[Bug] When using mysql-CDC Database synchronization Action On Combined Mode, mysql field changes cause tasks to wait indefinitely
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
master : 66cc214b
### Compute Engine
flink1.18
### Minimal reproduce step
1、run mysql-cdc database action combined mode:
'args'=' mysql_sync_database
--warehouse hdfs:///paimon/warehouse
--database ods_hobby_content
--mysql_conf hostname=***
--mysql_conf username=***
--mysql_conf password=***
--mysql_conf database-name=***
--mysql_conf server-id=10011-10021
--table_prefix ods_
--table_conf changelog-producer=input
--type_mapping to-nullable,tinyint1-not-bool
--table_conf bucket=3
--table_conf sink.parallelism=3
--table_conf sink.use-managed-memory-allocator=true
--table_conf deletion-vectors.enabled=true
--table_conf write-buffer-size=512m
--table_conf target-file-size=256m
--table_conf sink.managed.writer-buffer-memory=512m
--table_conf write-buffer-spillable=true
--table_conf sort-spill-threshold=10
--table_conf num-sorted-run.stop-trigger=2147483647
--table_conf lookup-wait=false
--mode combined
2、After running for a period of time, when the mysql table field changes, the flink log keeps output:
Field " + filedName + " not found. Waiting for schema update.
Field " + filedName + " not found. Waiting for schema update.
Field " + filedName + " not found. Waiting for schema update.
And it will continue until the flink job dies
I located the source code output location of the log:
org.apache.paimon.flink.sink.cdc.CdcRecordUtils#toGenericRow(:92line)

It seems that the field has been waiting for a change, and the data flow that handles the schema change has not been processed properly
Wonder if it is possible that the current topology never has time to process field changes when the data synchronization is being compressed, or that the schema change operator needs to add state?
### What doesn't meet your expectations?
no error
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at org.apache.paimon.flink.sink.cdc.CdcRecordUtils#toGenericRow, especially the field-not-found waiting path, then trace how schema changes move through the mysql-CDC database action in combined mode. Reproduce with the supplied Flink 1.18 configuration and a MySQL field change; done means the job processes the updated schema without waiting indefinitely or dying.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100