[lightning] TiDB Lightning replace can report success with corrupted functional and virtual generated indexes
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
**Environment**
TiDB master 05b396fb66; Lightning current source and a942e4684f nightly; real TiKV; one TiDB; MDL ON
**Steps**
Create a clustered-primary-key table with UNIQUE KEY uk_normalized_email ((LOWER(email))). Import CSV rows (1,Alice@example.com,protected-first) and (2,alice@example.com,replacement-second) using backend=local, add-index-by-sql=false, conflict.strategy=replace, checkpoint.enable=false, and checksum=required. After Lightning reports success, compare the base-table row set with FORCE INDEX(uk_normalized_email), then run ADMIN CHECK TABLE.
### 2. What did you expect to see? (Required)
Conflict replacement removes one logical duplicate before publishing success. Base and index scans return the same single row, and ADMIN CHECK TABLE succeeds.
### 3. What did you see instead? (Required)
On exact TiDB master 05b396fb66 with real TiKV, Lightning reports two conflicts and exits successfully. The base table retains both rows, the functional unique index returns only id=1, and ADMIN CHECK TABLE returns error 8223 for handle 2. An explicit VIRTUAL generated unique-key table has the same RED. Changing only VIRTUAL to STORED is GREEN. Recomputing generated columns before all three conflict-cleanup re-encode sites makes both product cases GREEN.
### 4. What is your TiDB version? (Required)
TiDB master 05b396fb66; Lightning current source and a942e4684f nightly; real TiKV; one TiDB; MDL ON
Likely root cause and fix direction
**Likely root cause**
ReplaceConflictKeys decodes persisted row values and calls BaseKVEncoder.AddRecord to reconstruct the row-owned KV set. DecodeRawRowData intentionally leaves virtual generated columns unset because they are not stored in the row. The conflict resolver never calls EvalGeneratedColumns, so it reconstructs a NULL or missing derived index key, fails to recognize the overwritten row as the owner of the conflicting index entry, and leaves that row behind after reporting successful resolution. Functional indexes use hidden virtual generated columns and reach the same path.
**Fix direction**
Centralize decode-plus-derived-column-recompute-plus-reencode in one helper and use it at every ReplaceConflictKeys reconstruction site. Pass the original import SessionOptions as well, so context-sensitive generated expressions are reconstructed under the same SQL mode and system variables. Add functional-index and explicit VIRTUAL/STORED conflict-replace regressions with base/index/admin parity.
Contributor guide
Research direction
Start at ReplaceConflictKeys and trace each reconstruction through DecodeRawRowData, BaseKVEncoder.AddRecord, and EvalGeneratedColumns, including all three cleanup sites and the original SessionOptions. Reproduce the functional-index and explicit VIRTUAL cases, then add regressions showing conflict replacement leaves matching base and index scans and a successful ADMIN CHECK TABLE.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100