pingcap / pingcap/tidb

[br] BR restore retry can report checksum success from a stale target checkpoint

Open
#70,121 0 comments 0 reactions 0 assignees View on GitHub
component/br found-by-ai may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/critical type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

**Environment**

TiDB/BR master 05b396fb66; one TiDB; real TiKV; MDL ON; default checkpoint enabled; explicit --checksum

**Steps**

Back up rows {1,2,3}. Run checkpoint restore with --checksum and fail after the correct table checksum has been persisted. While the task is paused, delete row 2 and insert row 99 through ordinary SQL. Retry the identical command with --checksum. Compare the backup and final target row sets and inspect skipped-kv-count-by-checkpoint plus checksum logs.

### 2. What did you expect to see? (Required)

A resumed --checksum restore either proves the target stayed write-fenced since the cached checksum, rescans current TiKV data, or fails closed after target drift.

### 3. What did you see instead? (Required)

On exact BR master 05b396fb66 with one TiDB, real TiKV, MDL ON, and default checkpointing, retry returns success, skips all 6 KVs, and logs success in validating checksum. The target remains {1,3,99}; backup row 2 is missing and row 99 was never backed up. A no-drift retry is GREEN. Forcing a current checksum scan rejects the same RED state with a checksum mismatch.

### 4. What is your TiDB version? (Required)

TiDB/BR master 05b396fb66; one TiDB; real TiKV; MDL ON; default checkpoint enabled; explicit --checksum

Likely root cause and fix direction

**Likely root cause**

InitCheckpoint loads a checksum keyed by downstream table ID. execAndValidateChecksum skips current TiKV checksum execution whenever that cached item exists and compares the historical item only with backup metadata. Command hash and table ID bind task identity but no target mutation epoch or write-fence proves that the cached checksum still describes current table contents.

**Fix direction**

Keep restored targets write-fenced across paused attempts, bind cached checksums to a target mutation epoch, or rescan current target data on every resumed --checksum validation. Do not treat command hash or table ID as a state-freshness proof.

Contributor guide

Open the contributing guide

Research direction

Start by tracing InitCheckpoint and execAndValidateChecksum in BR, then reproduce the paused restore with --checksum using the stated TiDB, TiKV, MDL, and checkpoint setup. Compare cached-checksum reuse with a forced current checksum scan. Done means a resumed restore cannot report success after target drift and either proves freshness, rescans the target, or fails closed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.