pingcap / pingcap/tidb

[crrf] after HA fault "TIKV and PD leader network partition" was resolved, the Pitr task could not resume automatically

Open
#69,141 2 comments 0 reactions 1 assignee Claimed by @RidRisR View on GitHub
component/br may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/critical type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

1. create cluster and enabled crrf
2. run workload
3. inject network_partition(latency:50ms)_tikv(all)_to_pd(leader)
4. reslove the HA fault
5. check pitr task status

### 2. What did you expect to see? (Required)
pitr task can be resume automatically

### 3. What did you see instead (Required)
```bash
sh-5.1# tiup br:v8.5.6 log status -u db-pd.tidb10319223607049118824:2379 --ca=/var/lib/cluster-client-tls/ca.crt --cert=/var/lib/cluster-client-tls/tls.crt --key=/var/lib/cluster-client-tls/tls.key
Starting component br: /root/.tiup/components/br/v8.5.6/br log status -u db-pd.tidb10319223607049118824:2379 --ca=/var/lib/cluster-client-tls/ca.crt --cert=/var/lib/cluster-client-tls/tls.crt --key=/var/lib/cluster-client-tls/tls.key
Detail BR log in /tmp/br.log.2026-06-12T05.38.17Z
● Total 1 Tasks.
> #1 <
name: log-781947
status: ○ ERROR
start: 2026-06-09 13:47:31 +0000
end: 2090-11-18 14:07:45.624 +0000
storage: s3://dbaas-dr-staging-us-west-2-1369847559694037350-487825215408/cluster-1031922360704911882
4-sr%28us-west-2%29-dr%28us-east-1%29/log-781947
speed(est.): 0.00 ops/s
checkpoint[global]: 2026-06-12 05:15:43.885 +0000; gap=22m35s
pause-time: 2026-06-12T05:35:57Z
pause-operator: ip-172-16-64-120.us-west-2.compute.internal
pause-operator-pid: 1
pause-payload[errcode]: KV:LogBackup:Unknown
pause-payload[errmesg]: [store = 1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [s
tore = 1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [sto
re = 1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store
= 1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store =
1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1
004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 100
4] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004]
Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] O
ther Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Oth
er Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other
Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other E
rror: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Err
or: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error
: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error:
[components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [c
omponents/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [com
ponents/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [compo
nents/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [compone
nts/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [component
s/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [components/
backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [components/ba
ckup-stream/src/endpoint.rs:284]: origin error: [store = 1004] retry encountered error, orig
in error is PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: Other E
rror: retry time exceeds for region Region { id: 6273, ver: 251, conf_ver: 6, range: [748000
0000000000FFD05F728000000000FF3EC9710000000000FA,7480000000000000FFD05F728000000000FF5C60710
000000000FA), peers: 1001,1004,1005,1014 }: PD client meet error: RpcFailure: 14-UNAVAILABLE
failed to connect to all addresses: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED D
eadline Exceeded: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all
addresses: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: PD clien
t meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all addresses: PD client meet
error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: PD client meet error: RpcFailure:
4-DEADLINE_EXCEEDED Deadline Exceeded: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED
Deadline Exceeded: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect to al
l addresses: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: PD cli
ent meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all addresses: PD client mee
t error: RpcFailure: 14-UNAVAILABLE failed to connect to all addresses: PD client meet error
: RpcFailure: 14-UNAVAILABLE failed to connect to all addresses: PD client meet error: RpcFa
ilure: 14-UNAVAILABLE failed to connect to all addresses: PD client meet error: RpcFailure:
14-UNAVAILABLE failed to connect to all addresses: PD client meet error: RpcFailure: 14-UNAV
AILABLE failed to connect to all addresses: PD client meet error: RpcFailure: 4-DEADLINE_EXC
EEDED Deadline Exceeded: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect
to all addresses: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all
addresses: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all address
es: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: PD client meet
error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded
error[store=1004]: KV:LogBackup:Unknown
error-happen-at[store=1004]: 2026-06-12 05:35:57.44 +0000; gap=2m21s
error-message[store=1004]: [store = 1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [s
tore = 1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [sto
re = 1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store
= 1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store =
1004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1
004] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 100
4] Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004]
Other Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] O
ther Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Oth
er Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other
Error: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other E
rror: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Err
or: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error
: [components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error:
[components/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [c
omponents/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [com
ponents/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [compo
nents/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [compone
nts/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [component
s/backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [components/
backup-stream/src/endpoint.rs:284]: origin error: [store = 1004] Other Error: [components/ba
ckup-stream/src/endpoint.rs:284]: origin error: [store = 1004] retry encountered error, orig
in error is PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: Other E
rror: retry time exceeds for region Region { id: 6273, ver: 251, conf_ver: 6, range: [748000
0000000000FFD05F728000000000FF3EC9710000000000FA,7480000000000000FFD05F728000000000FF5C60710
000000000FA), peers: 1001,1004,1005,1014 }: PD client meet error: RpcFailure: 14-UNAVAILABLE
failed to connect to all addresses: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED D
eadline Exceeded: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all
addresses: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: PD clien
t meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all addresses: PD client meet
error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: PD client meet error: RpcFailure:
4-DEADLINE_EXCEEDED Deadline Exceeded: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED
Deadline Exceeded: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect to al
l addresses: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: PD cli
ent meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all addresses: PD client mee
t error: RpcFailure: 14-UNAVAILABLE failed to connect to all addresses: PD client meet error
: RpcFailure: 14-UNAVAILABLE failed to connect to all addresses: PD client meet error: RpcFa
ilure: 14-UNAVAILABLE failed to connect to all addresses: PD client meet error: RpcFailure:
14-UNAVAILABLE failed to connect to all addresses: PD client meet error: RpcFailure: 14-UNAV
AILABLE failed to connect to all addresses: PD client meet error: RpcFailure: 4-DEADLINE_EXC
EEDED Deadline Exceeded: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect
to all addresses: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all
addresses: PD client meet error: RpcFailure: 14-UNAVAILABLE failed to connect to all address
es: PD client meet error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded: PD client meet
error: RpcFailure: 4-DEADLINE_EXCEEDED Deadline Exceeded
sh-5.1#
```
### 4. What is your TiDB version? (Required)

[v8.5.6-tidb-advancer-10586-dev]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.