apache / apache/paimon

[Bug] Incorrectly including tables matching excludingTablePattern in combined mode cdc

Open
#4,163 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

master

### Compute Engine

flink

### Minimal reproduce step

1. Create a database named cdc_test in mysql, then create primary table named pk_1, pk_2 ... pk_100 and non-primary table named non_pk_1, non_pk_2, ... non_pk_100.
2. Start a combined mode mysql database cdc job, and set excludingTablePattern to 'non_pk_.+'.
3. Concurently create non-primary table named non_pk_101 when starting cdc job.
4. Finally, the Jobmanager log will show "com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including ‘cdc_test.non_pk_101’ for further processing".

### What doesn't meet your expectations?

non_pk_101 obviously matches the excludingTablePattern and needs to be excluded. Since non_pk_101 is a non-primary key table, MySqlChunkSplitter will report an error:
Caused by: org.apache.flink.table.api.ValidationException: Incremental snapshot for tables requires primary key, but table cdc_test.non_pk_101 doesn't have primary key.

![image](https://github.com/user-attachments/assets/f5b72900-a8f7-44da-919b-316ee7e2d540)
combinedModeTableList func uses excluding pattern ?!(^db\\.tbl$)|(^...$),and this will miss the newly table that matches excludingTablePattern and created when starting cdc job.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the combinedModeTableList function and the TableDiscoveryUtils log path involved in MySQL CDC table discovery. Reproduce the concurrent creation of non_pk_101 with excludingTablePattern set, then verify that the matching table is excluded and MySqlChunkSplitter no longer reports the missing-primary-key error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.