pingcap / pingcap/tidb-binlog

drainer surport Synchronization exception skip

Open
#963 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request
Dominant language
Go
Stars
291
Forks
131
Avg merge
5m
Merged PRs (30d)
2

Description

Feature Request

drainer surport Synchronization exception skip
Is your feature request related to a problem? Please describe:
drainer surport Synchronization exception skip

Describe the feature you'd like:
When the synchronization is abnormal, the drainer exits directly. When the Draine is restarted, there are about 15 minutes of safe mode mode. At this time, there are two problems: 1. If safe mode can't solve this problem, you need to manually set ignore Tso, if the number of errors of the same type is very large, this is an immeasurable work, and at the same time it seriously affects the synchronization progress. Then on the production line, if one kind of errors are encountered, and the amount of data involved is very large, it is to skip this kind of data synchronization first, to ensure that other tables or this table do not conflict with the data synchronization, rather than the entire synchronization is blocked; 2 Even if the mode can be solved, after the safe mode period of time, if this problem occurs again, it needs to be restarted. If the safe mode is on all the time, it will consume performance and will not make sense, resulting in increased synchronization delay

Describe alternatives you've considered:
Maybe it's impossible for us to clearly distinguish what error causes synchronization interruption. Of course, drainer can record this error and feed it back to the monitor, but in drainer, we can distinguish between primary key conflict or non null fields. There are too many types of errors, and we hope to recover synchronization as soon as possible. Therefore, we can divide the error types into two types: 1. DDL error, 2. DML error, set two configuration items, whether to skip DDL / DML error or not. If yes is selected, the synchronization of this data will be skipped if DML / DDL error is encountered, and the synchronization of the next data will continue. If no is selected, the logic is the same as before. When an error is encountered, the drainer should record the corresponding error information, so as to find the cause and repair the data later
Teachability, Documentation, Adoption, Migration Strategy:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by tracing drainer synchronization error handling and safe-mode behavior, then determine how DDL and DML failures are distinguished. Done should include configurable skip behavior, continued synchronization, and recorded error details for later repair.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.