pingcap / pingcap/tidb

[br] BR snapshot restore can report success after concurrent DML creates persistent unique-index corruption

Open
#70,089 0 comments 0 reactions 0 assignees View on GitHub
component/br found-by-ai may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/critical type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

**Environment**

TiDB nightly ed2376acc6 and BR nightly a942e4684f on Classic real TiKV; MDL ON; target mode Normal; checksum default OFF

**Steps**

Back up t(id BIGINT PRIMARY KEY, u BIGINT UNIQUE, payload) containing (1,100001,...). Drop the target. Start ordinary BR restore table with enough duration (for example --ratelimit 1). As soon as BR creates empty target t, verify TIDB_TABLE_MODE=Normal and INSERT (1,900000000,'app-write-during-restore'). Wait for BR to report success. Compare COUNT/SUM using PRIMARY versus index u, query WHERE u=100001, and run ADMIN CHECK TABLE.

### 2. What did you expect to see? (Required)

BR keeps the physical restore target inaccessible until ingest and validation finish, or rejects concurrent target writes. A successful restore must preserve a one-to-one record/index mapping and every index lookup result must satisfy its predicate.

### 3. What did you see instead? (Required)

Official nightly BR exits 0 and reports 256000 restored KV. PRIMARY has 128000 rows, while unique index u has 128001 entries. WHERE u=100001 returns (1,900000000,'app-write-during-restore') with predicate u=100001 evaluating false. ADMIN CHECK TABLE reports index value 100001 differs from record value 900000000.

### 4. What is your TiDB version? (Required)

TiDB nightly ed2376acc6 and BR nightly a942e4684f on Classic real TiKV; MDL ON; target mode Normal; checksum default OFF

Likely root cause and fix direction

**Likely root cause**

Ordinary snapshot restore creates the target in TableModeNormal. TableModeRestore, which already blocks DML/DDL, is applied only to explicit-filter PiTR. SnapClient freezes TableInfo and rewrite rules after creation, and normal snapshot rewrite uses NewTimestamp=0, preserving backup MVCC timestamps. A newer application record therefore wins at the clustered key while the restored older unique-index key has no matching delete, leaving a persistent stale index entry. RestoreTables performs no target-mode/write-generation revalidation before SST ingest.

**Fix direction**

Create or transition every physical snapshot-restore target into TableModeRestore before it becomes visible, retain that fence through ingest/checksum/stats, and atomically return it to Normal only after success. For existing targets, acquire an equivalent generation/write lease and fail closed if the table ID, mode, or write epoch changes before ingest.

Contributor guide

Open the contributing guide

Research direction

Start at BR's RestoreTables path and trace how TableModeRestore, SnapClient's frozen TableInfo and rewrite rules, and NewTimestamp=0 behave for ordinary snapshot restore versus explicit-filter PiTR. Reproduce the concurrent-DML scenario, then verify that the target remains fenced through ingest and validation and that success leaves matching primary and unique-index records.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.