Lightning: Improve Fail-Fast Behavior for `max-error.conflict` Configuration
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
Currently, there is a configuration setting called `max-error.conflict` that restricts the maximum number of conflict errors for a Lightning task. However, currently this check is performed only when recording the conflict records into the meta table, after all the conflict data has already been constructed by the pebble DB and written into TiKV. If there are many conflicts, constructing all that conflict data can be very time-consuming. To address this, we propose that, during the construction of KV data, if the number of conflict records hits the `max-error.conflict` threshold, the entire import process should be halted and all the conflict records should be recorded into the meta table. This way, we can ensure that the `max-error.conflict` configuration can truly make the Lightning fail fast.
Contributor guide
Assessment
This issue has not been assessed yet.