pingcap / pingcap/tidb

log restore progress is stuck

Open
#60,160 1 comment 0 reactions 1 assignee Claimed by @3pointer View on GitHub
affects-9.0 component/br may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

- Large-scale cluster, data volume of 200TB, TiKV 100 nodes
- Large wide table
- do PiTR for this tidb cluster

### 2. What did you expect to see? (Required)
PiTR success

### 3. What did you see instead (Required)
log restore progress is stuck

![Image](https://github.com/user-attachments/assets/97a2dc6c-8467-441c-85a3-238c1be433a1)

[2025/03/18 12:34:27.972 +00:00] [WARN] [checkpoint.go:318] ["failed to flush checkpoint data"] [error="[tikv:9005]Region is unavailable"] [errorVerbose="[tikv:9005]Region is unavailable\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/errors.go:178\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/juju_adaptor.go:15\ngithub.com/pingcap/tidb/br/pkg/gluetidb.(*tidbSession).ExecuteInternal\n\t/workspace/source/tidb/br/pkg/gluetidb/glue.go:208\ngithub.com/pingcap/tidb/br/pkg/checkpoint.(*tableCheckpointStorage).flushCheckpointData\n\t/workspace/source/tidb/br/pkg/checkpoint/storage.go:151\ngithub.com/pingcap/tidb/br/pkg/checkpoint.(*CheckpointRunner[...]).doFlush\n\t/workspace/source/tidb/br/pkg/checkpoint/checkpoint.go:601\ngithub.com/pingcap/tidb/br/pkg/checkpoint.(*flusher[...]).doFlush\n\t/workspace/source/tidb/br/pkg/checkpoint/checkpoint.go:317\ngithub.com/pingcap/tidb/br/pkg/checkpoint.(*CheckpointRunner[...]).startCheckpointFlushLoop.func1\n\t/workspace/source/tidb/br/pkg/checkpoint/checkpoint.go:391\nruntime.goexit\n\t/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.7.linux-amd64/src/runtime/asm_amd64.s:1700"]
[2025/03/18 12:35:39.497 +00:00] [INFO] [region_request.go:970] ["throwing pseudo region error due to no replica available"] [req-ts=456733823589941250] [req-type=Prewrite] [region="{ region id: 1084, ver: 24375, confVer: 83 }"] [replica-read-type=leader] [stale-read=false] [request-sender="{rpcError:, replicaSelector: replicaSelectorV2{replicaReadType: leader, attempts: 14, cacheRegionIsValid: false, replicaStatus: [peer: 1401, store: 1039, isEpochStale: false, attempts: 10, attempts_time: 1.81ms, replica-epoch: 0, store-epoch: 0, store-state: resolved, store-liveness-state: reachable peer: 1725, store: 1539, isEpochStale: false, attempts: 1, attempts_time: 1.84ms, replica-epoch: 0, store-epoch: 0, store-state: resolved, store-liveness-state: reachable peer: 1744, store: 1503, isEpochStale: false, attempts: 1, attempts_time: 1.99ms, replica-epoch: 0, store-epoch: 0, store-state: resolved, store-liveness-state: reachable]}}"] [total-round-stats="{total-backoff: 1m11.5s, total-backoff-times: 11}"] [current-round-stats="{time: 1m11.5s, backoff: 1m11.5s, timeout: 30s, req-max-exec-timeout: 20s, retry-times: 13}"]

### 4. What is your TiDB version? (Required)

./br -V
Release Version: v9.0.0-beta.1.pre-433-g80d6b56
Git Commit Hash: 80d6b5683c5c9e655d1eab432b198d7fea9b7d5f
Git Branch: HEAD
Go Version: go1.23.7
UTC Build Time: 2025-03-18 02:44:07
Race Enabled: false

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.