apache / apache/paimon

[Feature] Optimize CDC sync database action to avoid blocking on listTables operation with large number of tables

Open
#5,955 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

When using CDC synchronous database operations on a database containing a large number of tables (hundreds of thousands), the "catalog. listTables (database)" operation in "SyncDataActionBase" may take a long time to complete, causing the entire synchronization job to start blocking for a long time. This will significantly affect the duration of CDC synchronization task initiation.

### Solution

### Current Behavior
The current implementation calls `catalog.listTables(database)` during initialization and maintains a `createdTables` set to track table creation status. This approach:
1. Blocks the entire sync process while listing all tables
2. Consumes unnecessary memory to maintain the `createdTables` set
3. Performs redundant operations when tables are created lazily

### Expected Behavior
The sync process should:
1. Avoid blocking on `listTables` operation during initialization
2. Create tables lazily when needed without maintaining a global `createdTables` set
3. Improve overall performance for databases with large numbers of tables

### Solution
Optimize the table creation logic by:
1. Removing the upfront `listTables` call in `SyncDatabaseActionBase`
2. Eliminating the `createdTables` set from `RichCdcMultiplexRecordEventParser`
3. Implementing lazy table creation in `CdcDynamicTableParsingProcessFunction` with existence checks

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Read SyncDatabaseActionBase and RichCdcMultiplexRecordEventParser first, then trace table creation through CdcDynamicTableParsingProcessFunction. Confirm the initialization path no longer blocks on catalog.listTables(database), global createdTables tracking is removed, and tables are created lazily with existence checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.