pingcap / pingcap/tidb

br: blocklist files written by older BR always fail checksum after #64465, breaking log truncate and repeat PITR

Open
#70,137 2 comments 0 reactions 0 assignees View on GitHub
affects-8.5 component/br severity/critical type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

**Affected branches**: master (unfixed as of 66be7a615b) and release-8.5 (≥ v8.5.6, via cherry-pick #65018). Introduced by #64465.

### What happened

`LogRestoreTableIDsBlocklistFile` (`br/pkg/restore/misc.go`) was changed incompatibly: proto field 2 (`snapshot_backup_ts`) was dropped and replaced by field 7 (`restore_start_ts`) — but the field is covered by the file checksum.

For any blocklist file written by an older BR:
1. field 2 is discarded as unknown on unmarshal, field 7 is absent → `RestoreStartTs = 0`;
2. the checksum is recomputed with 0 instead of the real `snapshot_backup_ts` → **always mismatches**;
3. `unmarshalLogRestoreTableIDsBlocklistFile` returns a hard `checksum mismatch ... may be corrupted` error, and `fastWalkLogRestoreTableIDsBlocklistFile` aborts the whole walk.

Two affected paths:
- **`br log truncate`** (`TruncateLogRestoreTableIDsBlocklistFiles`): any log-backup storage that ever had a PITR done by an older BR can no longer be truncated once `--until` passes the old file's commit TS. This is a routine retention operation — it breaks silently after a BR upgrade.
- **Repeat PITR pre-check** (`CheckTableTrackerContainsTableIDsFromBlocklistFiles`): restoring with `restored-ts >= old snapshotBackupTs` and `start-ts < old restoreCommitTs` reads the old file and fails.

### Reproduction (verified end-to-end)

Constructed a valid old-format blocklist file (fields 1/2/6/3/5 + legacy checksum) in a local storage and ran a current build of BR:

```
$ br log truncate --storage local:///tmp/s --until 400000000000000200 -y
Error: checksum mismatch (calculated checksum is w5qjCoQZFzVBX27Fwy9lm3FFOKIt8l7k+2NNbOrswbw= but the recorded checksum is rDbIQoFWoTwVKhDZIGAjPduBYwdD0otZBiQ66ybrrT0=), the log restore table IDs blocklist file may be corrupted
```

Cross-verified: the `recorded` value equals the legacy algorithm over the real `snapshot_backup_ts`, and the `calculated` value equals the new algorithm with `RestoreStartTs=0` — matching the error text character-for-character. The same file in the new format (field 7) truncates successfully.

### Suggested fix

Tolerate legacy files instead of hard-failing, e.g. fall back to the filename-parsed `(restoreCommitTs, S)` values with a warning (for truncate, deleting a file should not require trusting its contents), or keep proto field 2 reserved and fall back to the legacy checksum when `RestoreStartTs == 0`.

Contributor guide

Open the contributing guide

Research direction

Start in br/pkg/restore/misc.go, tracing unmarshalLogRestoreTableIDsBlocklistFile, fastWalkLogRestoreTableIDsBlocklistFile, and the truncate and repeat-PITR callers. Reproduce with the documented br log truncate command using a legacy blocklist file, then verify legacy files no longer abort checksum validation while current-format files still work.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.