pingcap / pingcap/tidb

[dr-autosync]PITR will be blocked if an online recovery is attempted while a log backup is in progress.

Open
#44,079 4 comments 0 reactions 1 assignee Claimed by @3pointer View on GitHub
affects-7.5 affects-8.1 affects-8.5 component/br may-affects-5.1 may-affects-5.2 may-affects-5.3 may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
After down primary dc and forcing a new cluster in the pd node located in the backup dc, do online recovery as following.
```

[2023/05/22 18:31:18.945 +08:00] [INFO] [drcluster.go:345] ["will run cmd"] [cmd:="tiup ctl:nightly pd -u http://pd3-peer.dr-autosync-hongmei-testbedwbxcc:2379/ unsafe remove-failed-stores 10,5,6"]
```

After 2min, online recovery succeeded.
```
[2023/05/22 18:33:21.168 +08:00] [INFO] [drcluster.go:785] ["will run cmd"] [cmd:="tiup ctl:nightly pd -u http://pd3-peer.dr-autosync-hongmei-testbedwbxcc:2379/ unsafe remove-failed-stores show"]
{
"info": "Unsafe recovery finished",
"time": "2023-05-22 18:32:45.698",
"details": [
"affected meta regions: 120",
"affected table ids: 14, 84, 86, 104, 106, 42, 91, 92, 50, 76, 90, 98, 18, 24, 58, 4, 44, 108, 281474976710651, 38, 52, 80, 116, 281474976710649, 40, 60, 281474976710652, 102, 281474976710654, 12, 46, 66, 74, 82, 8, 22, 78, 114, 48, 70, 95, 281474976710653, 54, 62, 56, 72, 97, 28, 93, 94, 110, 26, 30, 32, 68, 112, 281474976710650, 16, 20, 34, 64, 96, 6, 10, 36, 281474976710655"
]
}
```

### 2. What did you expect to see? (Required)

### 3. What did you see instead (Required)
pitr is blocked even after online recovery done.
```
tiup br:nightly log status --pd "pd3-peer:2379"
Starting component `br`: /root/.tiup/components/br/v7.2.0-alpha-nightly-20230521/br /root/.tiup/components/br/v7.2.0-alpha-nightly-20230521/br log status --pd pd3-peer:2379
Detail BR log in /tmp/br.log.2023-05-22T18.46.44+0800
● Total 1 Tasks.
> #1 <
name: pitr-20230522182606
status: ○ ERROR
start: 2023-05-22 18:26:30.04 +0800
end: 2090-11-18 22:07:45.624 +0800
storage: s3://tmp/log-backup-20230522182606
speed(est.): 0.00 ops/s
checkpoint[global]: 2023-05-22 18:28:02.54 +0800; gap=18m43s
error[store=1]: KV:LogBackup:RaftReq
error-happen-at[store=1]: 2023-05-22 18:32:35.751 +0800; gap=14m10s
error-message[store=1]: retry time exceeds: and error and error and error and error failed to get initial snapshot: failed to get the snapshot (region_id = 558): Error dur
ing requesting raftstore: message: "region 558 is in the recovery progress" recovery_in_progress { region_id: 558 }: failed to get initial snapshot
: failed to get the snapshot (region_id = 558): Error during requesting raftstore: message: "region 558 is in the recovery progress" recovery_in_pr
ogress { region_id: 558 }: failed to get initial snapshot: failed to get the snapshot (region_id = 558): Error during requesting raftstore: message
: "region 558 is in the recovery progress" recovery_in_progress { region_id: 558 }: failed to get initial snapshot: failed to get the snapshot (reg
ion_id = 558): Error during requesting raftstore: message: "region 558 is in the recovery progress" recovery_in_progress { region_id: 558 }: failed
to get initial snapshot: failed to get the snapshot (region_id = 558): Error during requesting raftstore: message: "region 558 is in the recovery
progress" recovery_in_progress { region_id: 558 }
```

### 4. What is your TiDB version? (Required)

master

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.