pingcap / pingcap/tidb

Lightning Dedup Introduces Fail-Fast Mechanism

Open
#40,743 4 comments 0 reactions 0 assignees View on GitHub
component/lightning type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
Currently, when doing de-dup in Lightning, it will scan the table records and handle ALL of them. This is a time-consuming job if there are so many duplicated rows after importing KVs into TiDB from Lightning ( For example, choose the wrong columns as unique keys or hit [this bug](https://github.com/pingcap/tidb/issues/39477) ) . In these situations, resolving ALL the duplicated records is not a sensible idea.
If there are quite many duplicated records to be resolved, it idicates that there are some problems with the data itself. Lightning had better fail fast, and let users check the data, thus saving the de-dup time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.