[Feature] Optimize CDC sync database action to avoid blocking on listTables operation with large number of tables
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
When using CDC synchronous database operations on a database containing a large number of tables (hundreds of thousands), the "catalog. listTables (database)" operation in "SyncDataActionBase" may take a long time to complete, causing the entire synchronization job to start blocking for a long time. This will significantly affect the duration of CDC synchronization task initiation.
### Solution
### Current Behavior
The current implementation calls `catalog.listTables(database)` during initialization and maintains a `createdTables` set to track table creation status. This approach:
1. Blocks the entire sync process while listing all tables
2. Consumes unnecessary memory to maintain the `createdTables` set
3. Performs redundant operations when tables are created lazily
### Expected Behavior
The sync process should:
1. Avoid blocking on `listTables` operation during initialization
2. Create tables lazily when needed without maintaining a global `createdTables` set
3. Improve overall performance for databases with large numbers of tables
### Solution
Optimize the table creation logic by:
1. Removing the upfront `listTables` call in `SyncDatabaseActionBase`
2. Eliminating the `createdTables` set from `RichCdcMultiplexRecordEventParser`
3. Implementing lazy table creation in `CdcDynamicTableParsingProcessFunction` with existence checks
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Read SyncDatabaseActionBase and RichCdcMultiplexRecordEventParser first, then trace table creation through CdcDynamicTableParsingProcessFunction. Confirm the initialization path no longer blocks on catalog.listTables(database), global createdTables tracking is removed, and tables are created lazily with existence checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100