pingcap / pingcap/tidb

[br] BR abort can delete a live restore because its row lock suppresses heartbeats

Open
#70,033 0 comments 0 reactions 0 assignees View on GitHub
component/br found-by-ai may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/critical type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

> Evidence status: confirmed.

### 1. Minimal reproduce step (Required)

On current source, create a running restore registry entry and continuously call UpdateHeartbeat from its dedicated heartbeat session. Concurrently call FindAndDeleteMatchingTask for the same configuration. A deterministic real-TiKV test shortens only the one-minute ticker and five-check count. Before the abort transaction, heartbeat advances and the task is classified active. After abort locks the row with SELECT FOR UPDATE, heartbeat UPDATE gets kv:9007 write conflict; both observations stay unchanged; abort returns the task ID and the row count becomes zero. This RED repeated 3/3. A no-heartbeat stale control deleted the row successfully 3/3.

### 2. What did you expect to see? (Required)

A matching restore with an active heartbeat writer must not be deleted; abort should return zero and retain the registry row.

### 3. What did you see instead? (Required)

The abort transaction holds the matching registry row lock while waiting for heartbeat changes. That same lock prevents the independent heartbeat session from updating, so the observer manufactures the stale evidence it relies on and then deletes the live task.

### 4. What is your TiDB version? (Required)

current master 13282a8bd06b

Likely root cause and fix direction

FindAndDeleteMatchingTask starts one pessimistic transaction, locks the row with FOR UPDATE, and calls isTaskStale inside that transaction. UpdateHeartbeat writes last_heartbeat_time on the same row through another session and conflicts with the abort lock. After the unchanged window, deletion uses only task ID.

Contributor guide

Open the contributing guide

Research direction

Start by locating FindAndDeleteMatchingTask and UpdateHeartbeat, then run the deterministic real-TiKV reproduction described in the issue. Trace the transaction and row-lock behavior around isTaskStale. Done means an active heartbeat prevents deletion and returns zero, while a genuinely stale task is still deleted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.