drainer surport Synchronization exception skip
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 291
- Forks
- 131
- Avg merge
- 5m
- Merged PRs (30d)
- 2
Description
Feature Request
drainer surport Synchronization exception skip
Is your feature request related to a problem? Please describe:
drainer surport Synchronization exception skip
Describe the feature you'd like:
When the synchronization is abnormal, the drainer exits directly. When the Draine is restarted, there are about 15 minutes of safe mode mode. At this time, there are two problems: 1. If safe mode can't solve this problem, you need to manually set ignore Tso, if the number of errors of the same type is very large, this is an immeasurable work, and at the same time it seriously affects the synchronization progress. Then on the production line, if one kind of errors are encountered, and the amount of data involved is very large, it is to skip this kind of data synchronization first, to ensure that other tables or this table do not conflict with the data synchronization, rather than the entire synchronization is blocked; 2 Even if the mode can be solved, after the safe mode period of time, if this problem occurs again, it needs to be restarted. If the safe mode is on all the time, it will consume performance and will not make sense, resulting in increased synchronization delay
Describe alternatives you've considered:
Maybe it's impossible for us to clearly distinguish what error causes synchronization interruption. Of course, drainer can record this error and feed it back to the monitor, but in drainer, we can distinguish between primary key conflict or non null fields. There are too many types of errors, and we hope to recover synchronization as soon as possible. Therefore, we can divide the error types into two types: 1. DDL error, 2. DML error, set two configuration items, whether to skip DDL / DML error or not. If yes is selected, the synchronization of this data will be skipped if DML / DDL error is encountered, and the synchronization of the next data will continue. If no is selected, the logic is the same as before. When an error is encountered, the drainer should record the corresponding error information, so as to find the cause and repair the data later
Teachability, Documentation, Adoption, Migration Strategy:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing drainer synchronization error handling and safe-mode behavior, then determine how DDL and DML failures are distinguished. Done should include configurable skip behavior, continued synchronization, and recorded error details for later repair.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100