[Bug] Issue with job recovery from savepoint when topology changes in CDC ingestion
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
0.5-SNAPSHOT
### Compute Engine
Flink
### Minimal reproduce step
When restarting a job from a savepoint in the Apache Flink CDC (Change Data Capture) engine, it is crucial to consider the impact of topology changes and the proper handling of hash UIDs for state recovery.
For instance, let's consider a CDC job that captures changes from a source database and writes them to a target sink. Suppose the initial savepoint is taken when the job processes tables A and B from the source database. Now, if a new table C is added to the source database and the job is restarted from the savepoint, the absence of hash UIDs for the operators involved in processing table C can result in state unavailability.
To address this issue effectively, it is necessary to determine which states should be ignored or skipped when new tables are introduced. By properly configuring the hash UIDs for all affected operators, the CDC engine can ensure that the state associated with each operator is correctly identified and recovered during the restart process. This ensures the reliability and completeness of the CDC job, even when there are changes in the job's topology.
Please note that this issue specifically pertains to the Apache Flink CDC engine and its handling of job recovery from savepoints when there are changes in the job's topology.
### What doesn't meet your expectations?
Now Paimon does not assign UIDs to tables. Note that UIDs should relate to both database and tables, or there might be conflict when there are two tables with same name but in different database.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Apache Flink CDC job topology and how Paimon assigns operator UIDs for source tables during savepoint recovery. Reproduce recovery after adding table C, including same-named tables in different databases, and verify that state is skipped or recovered correctly without UID conflicts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100