apache / apache/paimon

[cdc] optimize database sync table filter logic

Open
#3,654 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

master

### Compute Engine

flink

### Minimal reproduce step

none

### What doesn't meet your expectations?

In the current database_sync logic, if the including_tables or excluding_tables parameters are set, a series of parsing and conversion logic will still be performed on the tables that need to be filtered. We should filter them as soon as we can get the tableName to avoid some meaningless parsing and conversion operations.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the database_sync implementation and trace how including_tables and excluding_tables are applied after tableName becomes available. Confirm where parsing and conversion occur, then verify that filtered tables bypass those operations and run the relevant database-sync tests or checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.