apache / apache/doris-flink-connector

[Bug] mysql-sync-database should skip no-PK tables in initial snapshot mode or provide clear guidance

Open
#641 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Description

## Bug Description

When using `mysql-sync-database` for full-database real-time sync, tables without primary keys are not filtered automatically.

In `initial` startup mode (incremental snapshot phase), these no-PK tables may fail split/chunk-based snapshot reading, causing the Flink job to repeatedly fail and restart.

## Version

- Doris Flink Connector: master (or latest snapshot)
- Flink: 1.17.x
- Source DB: MySQL
- Runtime: Flink CDC incremental snapshot mode (`scan.startup.mode=initial`)

## Reproduction Steps

1. Prepare a MySQL database with mixed tables:
- table A: has primary key
- table B: no primary key
2. Run `mysql-sync-database` with broad table pattern (e.g. `including-tables=".*"`).
3. Use startup mode `initial` and do not configure chunk key for no-PK table.

## Expected Behavior

- Connector should not include no-PK tables in snapshot sync by default, **or**
- provide explicit and clear guidance/error before job starts unstable retries.
- If user explicitly configures chunk key for a no-PK table, that table should still be syncable.

## Actual Behavior

- no-PK tables are included in source table list;
- snapshot split/chunk phase fails;
- job keeps failing/restarting.

## Logs / Error (example)

```text
(snapshot split read failure on no primary key table)
(job restart loop)

### Solution

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Begin with the mysql-sync-database path for scan.startup.mode=initial and trace how tables enter incremental snapshot splitting. Compare tables with and without primary keys, including an explicitly configured chunk key; done means the chosen default or guidance prevents restart loops while preserving the explicit case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.