[lightning] TiDB Lightning can report success after pagination skips replace conflicts
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
This reproduces on TiDB/Lightning master
`05b396fb6636f73b3bc06b09107cf43f2c725c35` with one TiDB, one PD, real TiKV,
MDL enabled, strict SQL mode, the local backend, and
`conflict.strategy = "replace"`.
Set `LIGHTNING` to the current `tidb-lightning` binary and run:
```bash
set -euo pipefail
LIGHTNING=${LIGHTNING:-tidb-lightning}
MYSQL=${MYSQL:-mysql}
HOST=${HOST:-127.0.0.1}
PORT=${PORT:-4000}
STATUS_PORT=${STATUS_PORT:-10080}
PD=${PD:-127.0.0.1:2379}
DB=lightning_page_repro
TASK_DB=lightning_page_repro_tasks
WORK=$(mktemp -d /tmp/lightning-page-repro.XXXXXX)
DATA_A=$WORK/data-a
DATA_B=$WORK/data-b
mkdir -p "$DATA_A" "$DATA_B"
mysql_cmd=("$MYSQL" -h"$HOST" -P"$PORT" -uroot -N -B)
"${mysql_cmd[@]}" -e "
DROP DATABASE IF EXISTS \`$DB\`;
DROP DATABASE IF EXISTS \`$TASK_DB\`;"
for dir in "$DATA_A" "$DATA_B"; do
printf 'CREATE DATABASE `%s`;\n' "$DB" >"$dir/$DB-schema-create.sql"
cat >"$dir/$DB.accounts-schema.sql" <<'SQL'
CREATE TABLE accounts (
id BIGINT PRIMARY KEY CLUSTERED,
u BIGINT NOT NULL,
payload VARCHAR(64) NOT NULL,
UNIQUE KEY uk_u(u)
);
SQL
done
# Task A leaves exactly 999 retained index-conflict rows.
printf 'id,u,payload\n' >"$DATA_A/$DB.accounts.0.csv"
for i in $(seq 1 999); do
if (( i <= 996 )); then
u=$(((i + 1) / 2))
else
u=499
fi
printf '%d,%d,history-%04d\n' "$i" "$u" "$i" \
>>"$DATA_A/$DB.accounts.0.csv"
done
cat >"$DATA_B/$DB.accounts.0.csv" <<'CSV'
id,u,payload
2001,1001,current-first
2002,1001,current-second
CSV
write_config() {
path=$1
data_dir=$2
sorted_dir=$3
cat >"$path" <
Likely root cause and fix direction
`conflict_error_v4` stores `task_id`, but `ReplaceConflictKeys` queries rows by
table name and KV type without scoping them to the current task. Its page size
is 1000, and both conflict loops use:
```go
if len(handleKeys) == 0 {
break
}
```
`handleKeys` is the effect produced by a page, not proof that the input is
exhausted. The loop should count rows read, stop only on an empty input page,
advance the cursor after every nonempty page, and scope resolution to the
current `task_id`.
Contributor guide
Research direction
Start with the conflict_error_v4 records and the ReplaceConflictKeys entry point described in the issue, then run the supplied pagination reproduction against TiDB Lightning. Trace both conflict-resolution loops and compare rows read with effect rows produced. Done means pagination processes all current conflict rows, scopes resolution to the current task, and the final ADMIN CHECK TABLE passes without Lightning reporting false success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 66/100