Support managed initial snapshots in the TiDB connector
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Follow up to #789. In the discussion there the maintainers agreed the TiDB connector should provide snapshot support through TiDB's MySQL compatible SQL endpoint, see also the review of debezium/debezium-connector-tidb#3.
The first iteration of the connector only streams the Debezium format output of a TiCDC changefeed. It takes no data snapshot, so a new connector never captures rows that existed before the changefeed.
Proposal:
* Add snapshot modes `initial` and `initial_only` next to the default `no_data`.
* Read the current TSO once with `SHOW MASTER STATUS`, pin the session with the `tidb_snapshot` variable and read every captured table at that one consistent point over JDBC. No locks are needed. The TSO must stay within the GC lifetime of the cluster for the duration of the snapshot.
* Store the snapshot TSO in the offsets together with the standard snapshot state keys. During streaming, drop events whose `commit_ts` is not newer than the snapshot TSO, so a changefeed whose `start-ts` lies at or before the snapshot TSO hands off to streaming without duplicates.
* Map snapshot rows to the same Kafka Connect schemas TiCDC uses in its Debezium output, so snapshot events and streamed events describe tables the same way.
* Only require the `database.hostname`, `database.port`, `database.user` and `database.password` settings when the selected snapshot mode captures data.
Incremental snapshots are out of scope for this issue. They need the signal mechanism and write access to a signal table and can be tracked separately.
Contributor guide
Assessment
This issue has not been assessed yet.