pingcap / pingcap/tidb

Lightning: Improve Fail-Fast Behavior for `max-error.conflict` Configuration

Open
#42,471 2 comments 0 reactions 0 assignees View on GitHub
component/lightning type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
Currently, there is a configuration setting called `max-error.conflict` that restricts the maximum number of conflict errors for a Lightning task. However, currently this check is performed only when recording the conflict records into the meta table, after all the conflict data has already been constructed by the pebble DB and written into TiKV. If there are many conflicts, constructing all that conflict data can be very time-consuming. To address this, we propose that, during the construction of KV data, if the number of conflict records hits the `max-error.conflict` threshold, the entire import process should be halted and all the conflict records should be recorded into the meta table. This way, we can ensure that the `max-error.conflict` configuration can truly make the Lightning fail fast.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.