debezium / debezium/dbz

Support managed initial snapshots in the TiDB connector

Open
#2,439 0 comments 0 reactions 0 assignees View on GitHub
component/tidb-connector type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Follow up to #789. In the discussion there the maintainers agreed the TiDB connector should provide snapshot support through TiDB's MySQL compatible SQL endpoint, see also the review of debezium/debezium-connector-tidb#3.

The first iteration of the connector only streams the Debezium format output of a TiCDC changefeed. It takes no data snapshot, so a new connector never captures rows that existed before the changefeed.

Proposal:

* Add snapshot modes `initial` and `initial_only` next to the default `no_data`.
* Read the current TSO once with `SHOW MASTER STATUS`, pin the session with the `tidb_snapshot` variable and read every captured table at that one consistent point over JDBC. No locks are needed. The TSO must stay within the GC lifetime of the cluster for the duration of the snapshot.
* Store the snapshot TSO in the offsets together with the standard snapshot state keys. During streaming, drop events whose `commit_ts` is not newer than the snapshot TSO, so a changefeed whose `start-ts` lies at or before the snapshot TSO hands off to streaming without duplicates.
* Map snapshot rows to the same Kafka Connect schemas TiCDC uses in its Debezium output, so snapshot events and streamed events describe tables the same way.
* Only require the `database.hostname`, `database.port`, `database.user` and `database.password` settings when the selected snapshot mode captures data.

Incremental snapshots are out of scope for this issue. They need the signal mechanism and write access to a signal table and can be tracked separately.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.