apache / apache/doris-flink-connector
[Bug] mysql-sync-database should skip no-PK tables in initial snapshot mode or provide clear guidance
- Dominant language
- Java
- Stars
- 385
- Forks
- 279
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 6
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
## Bug Description
When using `mysql-sync-database` for full-database real-time sync, tables without primary keys are not filtered automatically.
In `initial` startup mode (incremental snapshot phase), these no-PK tables may fail split/chunk-based snapshot reading, causing the Flink job to repeatedly fail and restart.
## Version
- Doris Flink Connector: master (or latest snapshot)
- Flink: 1.17.x
- Source DB: MySQL
- Runtime: Flink CDC incremental snapshot mode (`scan.startup.mode=initial`)
## Reproduction Steps
1. Prepare a MySQL database with mixed tables:
- table A: has primary key
- table B: no primary key
2. Run `mysql-sync-database` with broad table pattern (e.g. `including-tables=".*"`).
3. Use startup mode `initial` and do not configure chunk key for no-PK table.
## Expected Behavior
- Connector should not include no-PK tables in snapshot sync by default, **or**
- provide explicit and clear guidance/error before job starts unstable retries.
- If user explicitly configures chunk key for a no-PK table, that table should still be syncable.
## Actual Behavior
- no-PK tables are included in source table list;
- snapshot split/chunk phase fails;
- job keeps failing/restarting.
## Logs / Error (example)
```text
(snapshot split read failure on no primary key table)
(job restart loop)
### Solution
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Begin with the mysql-sync-database path for scan.startup.mode=initial and trace how tables enter incremental snapshot splitting. Compare tables with and without primary keys, including an explicitly configured chunk key; done means the chosen default or guidance prevents restart loops while preserving the explicit case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100