pingcap / pingcap/tidb

When BR failed to restore an SST file, it should report the path of that SST file as well as the table it belongs to

Open
#60,312 0 comments 0 reactions 0 assignees View on GitHub
component/br type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

```bash
# prepare the data
mysql -u root -h 127.0.0.1 -P 4000 -e 'drop schema if exists test; create schema test;'
for i in $(seq 0 9); do mysql -u root -h 127.0.0.1 -P 4000 -e "create table test.t$i(a int primary key, b double not null); insert into test.t$i with recursive a as (select 1 c union all select c+1 from a where c < 100) select c, rand() from a;"; done

# backup
tiup br backup db --db test -s 'local:///tmp/br-corruption-test/'

# corrupt one sst file
dd if=/dev/zero of="$(ls -1 /tmp/br-corruption-test/*/*.sst | head -1)" bs=512 seek=2 count=1 conv=notrunc

# restore (warning: will take VERY LONG TIME (10 minutes) due to retries)
mysql -u root -h 127.0.0.1 -P 4000 -e 'drop schema test;'
tiup br restore full -s 'local:///tmp/br-corruption-test/' --log-file '/tmp/br-corruption-test.restore.log'
```

### 2. What did you expect to see? (Required)

The restore log clearly indicates which SST file is corrupted and which table does this SST file belongs to.

### 3. What did you see instead (Required)

The error message refers to the temporary filename downloaded into TiKV:

```
Error:
Engine Engine(Status { code: IoError, sub_code: None, sev: NoError, state: "Corruption: block checksum mismatch: stored = 472943280, computed = 3090244675, type = 1 in ~tikv/data/import/.temp/66cfaa37-339f-4bf7-b1ac-eac266a84e0b_154_1_73_write_2.sst offset 1416 size 1412" }): [BR:KV:ErrKVDownloadFailed]download sst failed;
Engine Engine(Status { code: IoError, sub_code: None, sev: NoError, state: "Corruption: block checksum mismatch: stored = 472943280, computed = 3090244675, type = 1 in ~tikv/data/import/.temp/ec1d4618-a71f-4bea-a5b0-416f143a72d6_154_1_73_write_2.sst offset 1416 size 1412" }): [BR:KV:ErrKVDownloadFailed]download sst failed;
<125 lines snipped>
Engine Engine(Status { code: IoError, sub_code: None, sev: NoError, state: "Corruption: block checksum mismatch: stored = 472943280, computed = 3090244675, type = 1 in ~tikv/data/import/.temp/bef05ec8-b8eb-4428-a99f-84117d884683_154_1_73_write_2.sst offset 1416 size 1412" }): [BR:KV:ErrKVDownloadFailed]download sst failed
```

The only way to get back the real SST file name is to search the TiKV logs:

```console
$ grep 66cfaa37-339f-4bf7-b1ac-eac266a84e0b_154_1_73_write_2 ~tikv/tikv.log
[2025/03/28 12:33:52.314 +08:00] [ERROR] [sst_importer.rs:418] ["download failed"] [err_code=KV:Engine:Engine] [err="Engine Engine(Status { code: IoError, sub_code: None, sev: NoError, state: \"Corruption: block checksum mismatch: stored = 472943280, computed = 3090244675, type = 1 in ~tikv/data/import/.temp/66cfaa37-339f-4bf7-b1ac-eac266a84e0b_154_1_73_write_2.sst offset 1416 size 1412\" })"] [name=1/136_64_27a057d447fb174bf1f1b07bd6bcc7b4d84560e115d087b35e5d1a5febf484d0_1743135462105_write.sst] [meta="uuid: 66CFAA37339F4BF7B1ACEAC266A84E0B range { start: 7480000000000000985F72 end: 7480000000000000985F72FFFFFFFFFFFFFFFFFFFF } length: 2972 cf_name: \"write\" region_id: 154 region_epoch { conf_ver: 1 version: 73 } cipher_iv: AA2E00D2EFD334D3E392B2D719BCC80E"] [thread_id=156]
```

Even so it is very hard to find which table does `136_64_«uuid»_1743135462105_write.sst` belong to, short of reading `backupmeta` manually.

### 4. What is your TiDB version? (Required)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.