pingcap / pingcap/tidb-binlog

Correct the implemention of drainer mysql checkpoint

Open
#889 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request
Dominant language
Go
Stars
291
Forks
131
Avg merge
5m
Merged PRs (30d)
2

Description

Feature Request

Is your feature request related to a problem? Please describe:

There are some problems with implemention of drainer mysql checkpoint, one mysql checkpoint example:

mysql> select * from tidb_binlog.checkpoint;
+---------------------+---------------------------------------------------------------------------------------------------------+
| clusterID           | checkPoint                                                                                              |
+---------------------+---------------------------------------------------------------------------------------------------------+
| 6782769820199954307 | {"commitTS":413989456971300865,"ts-map":{"master-ts":413989358457585669,"slave-ts":413989440066945029}} |
| 6782784420681659503 | {"commitTS":414399178233348097,"ts-map":{"master-ts":414398077844324967,"slave-ts":414398082560032770}} |
+---------------------+---------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec

there're two columns in checkpoint table

  • clusterID which is primary key
  • checkpoint which contain the specific checkpoint value
    , and clusterID is fetched from PD service of theupstream tidb cluster

the implementation brings three problems:

  • how to fetch clusterID while the upstream cluster is completely down
  • if user want to deplay multiple drainer to replicate different tables from one upstream tidb cluster to the same downstream, they must specify different checkpoint schema and table. It's unreasonable, and user can't always figure out the weird logic

Describe the feature you'd like:

maybe we let user to specify the node-id for every drainer, and store node-id into checkpoint

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests. Start by locating drainer's MySQL checkpoint handling and its PD clusterID lookup, then examine how upstream unavailability and multiple drainers sharing one downstream are handled. Done should be an agreed checkpoint identity and storage design that addresses both cases, with tests covering them.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.