pingcap / pingcap/tidb

br: clean restore checkpoint tables failed will not lead to restore failure or error report

Open
#60,233 2 comments 0 reactions 0 assignees View on GitHub
component/br severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1. restore to cluster

### 2. What did you expect to see? (Required)
no error
### 3. What did you see instead (Required)
console shows restore success
```
Detail BR log in /tmp/br.log.2025-03-21T08.52.26Z
[2025/03/21 10:49:16.921 +00:00] [INFO] [collector.go:77] ["Full Restore success summary"] [total-ranges=8637] [ranges-succeed=8637] [range
s-failed=0] [restore-files=1h39m15.105158713s] [restore-pipeline=14m50.439864315s] [merge-ranges=72.253028ms] [split-regions=29.436590878s]
[default-CF-files=0] [write-CF-files=8637] [split-keys=8633] [total-take=1h56m50.68558604s] [RestoreTS=456798305776828556] [total-kv=12000
000000] [total-kv-size=1.524TB] [average-speed=217.4MB/s] [restore-data-size(after-compressed)=657.4GB] [Size=657422234024] [BackupTS=45498
0106477371394]
```
but br log shows that checkpoint data are not cleaned successfully, and console doesn't report warning or error about it
```
[2025/03/21 11:00:38.253 +00:00] [WARN] [restore.go:846] ["failed to remove checkpoint data for snapshot restore"] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [errorVerbose="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/errors.go:178\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/executor.(*DDLExec).toErr\n\t/workspace/source/tidb/pkg/executor/ddl.go:74\ngithub.com/pingcap/tidb/pkg/executor.(*DDLExec).Next\n\t/workspace/source/tidb/pkg/executor/ddl.go:234\ngithub.com/pingcap/tidb/pkg/executor/internal/exec.Next\n\t/workspace/source/tidb/pkg/executor/internal/exec/executor.go:460\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).next\n\t/workspace/source/tidb/pkg/executor/adapter.go:1269\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor\n\t/workspace/source/tidb/pkg/executor/adapter.go:1018\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay\n\t/workspace/source/tidb/pkg/executor/adapter.go:851\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec\n\t/workspace/source/tidb/pkg/executor/adapter.go:614\ngithub.com/pingcap/tidb/pkg/session.runStmt\n\t/workspace/source/tidb/pkg/session/session.go:2305\ngithub.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt\n\t/workspace/source/tidb/pkg/session/session.go:2167\ngithub.com/pingcap/tidb/pkg/session.(*session).ExecuteInternal\n\t/workspace/source/tidb/pkg/session/session.go:1540\ngithub.com/pingcap/tidb/br/pkg/gluetidb.(*tidbSession).ExecuteInternal\n\t/workspace/source/tidb/br/pkg/gluetidb/glue.go:210\ngithub.com/pingcap/tidb/br/pkg/checkpoint.dropCheckpointTables\n\t/workspace/source/tidb/br/pkg/checkpoint/storage.go:331\ngithub.com/pingcap/tidb/br/pkg/checkpoint.(*TableMetaManager[...]).RemoveCheckpointData\n\t/workspace/source/tidb/br/pkg/checkpoint/manager.go:201\ngithub.com/pingcap/tidb/br/pkg/task.RunRestore\n\t/workspace/source/tidb/br/pkg/task/restore.go:844\nmain.runRestoreCommand\n\t/workspace/source/tidb/br/cmd/br/restore.go:80\nmain.newFullRestoreCommand.func1\n\t/workspace/source/tidb/br/cmd/br/restore.go:186\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041\nmain.main\n\t/workspace/source/tidb/br/cmd/br/main.go:37\nruntime.main\n\t/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.7.linux-amd64/src/runtime/proc.go:272\nruntime.goexit\n\t/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.7.linux-amd64/src/runtime/asm_amd64.s:1700"]
```
### 4. What is your TiDB version? (Required)
master
v9.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.