pingcap / pingcap/tidb

br restore failed with error "Coprocessor task terminated due to exceeding the deadline"

Open
#59,726 1 comment 0 reactions 1 assignee Claimed by @BornChanger View on GitHub
affects-6.5 component/br may-affects-6.1 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1、br restore

### 2. What did you expect to see? (Required)
br restore can succeed

### 3. What did you see instead (Required)
br restore failed with error "Coprocessor task terminated due to exceeding the deadline"

`[2025/02/22 00:42:42.626 +00:00] [ERROR] [advancer.go:406] ["listen task meet error, would reopen."] [error=EOF] [stack="github.com/pingcap/tidb/br/pkg/streamhelper.(*CheckpointAdvancer).StartTaskListener.func1\n\t/workspace/source/tidb/br/pkg/streamhelper/advancer.go:406"]
[2025/02/22 00:42:42.626 +00:00] [INFO] [domain.go:586] ["topNSlowQueryLoop exited."]
[2025/02/22 00:42:42.626 +00:00] [INFO] [advancer.go:409] ["[log backup advancer] Task watcher exits due to some error."] [error=EOF]
[2025/02/22 00:42:42.626 +00:00] [INFO] [manager.go:258] ["failed to campaign"] ["owner info"="[log-backup] /tidb/br-stream/owner ownerManager 4a2ebeb9-06a8-4342-821d-3bce023cb2f0"] [error="context canceled"]
[2025/02/22 00:42:42.626 +00:00] [INFO] [manager.go:230] ["etcd session is done, creates a new one"] ["owner info"="[log-backup] /tidb/br-stream/owner ownerManager 4a2ebeb9-06a8-4342-821d-3bce023cb2f0"]
[2025/02/22 00:42:42.626 +00:00] [INFO] [owner_daemon.go:85] ["daemon loop exits"] [id=4a2ebeb9-06a8-4342-821d-3bce023cb2f0] [daemon-id=LogBackup::Advancer]
[2025/02/22 00:42:42.626 +00:00] [INFO] [manager.go:234] ["break campaign loop, NewSession failed"] ["owner info"="[log-backup] /tidb/br-stream/owner ownerManager 4a2ebeb9-06a8-4342-821d-3bce023cb2f0"] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tidb/util.contextDone\n\t/workspace/source/tidb/util/etcd.go:90\ngithub.com/pingcap/tidb/util.NewSession\n\t/workspace/source/tidb/util/etcd.go:50\ngithub.com/pingcap/tidb/owner.(*ownerManager).campaignLoop\n\t/workspace/source/tidb/owner/manager.go:232\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"]
[2025/02/22 00:42:42.626 +00:00] [INFO] [domain.go:932] ["domain closed"] ["take time"=744.79831ms]
[2025/02/22 00:42:42.626 +00:00] [INFO] [client.go:783] ["[pd] stop fetching the pending tso requests due to context canceled"] [dc-location=global]
[2025/02/22 00:42:42.626 +00:00] [INFO] [client.go:719] ["[pd] exit tso dispatcher"] [dc-location=global]
[2025/02/22 00:42:42.626 +00:00] [INFO] [base_client.go:143] ["[pd] exit member loop due to context canceled"]
[2025/02/22 00:42:42.626 +00:00] [INFO] [pd.go:211] ["closed pd http client"]
[2025/02/22 00:42:42.626 +00:00] [INFO] [collector.go:220] ["units canceled"] [cancel-unit=0]
[2025/02/22 00:42:42.626 +00:00] [INFO] [collector.go:74] ["DataBase Restore failed summary"] [total-ranges=15867] [ranges-succeed=15867] [ranges-failed=0] [split-region=1m6.711664968s] [restore-ranges=15803]
[2025/02/22 00:42:42.626 +00:00] [ERROR] [restore.go:59] ["failed to restore"] [error="context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled"] [errorVerbose="the following errors occurred:\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tidb/br/pkg/task.runRestore\n\t/workspace/source/tidb/br/pkg/task/restore.go:849\ngithub.com/pingcap/tidb/br/pkg/task.RunRestore\n\t/workspace/source/tidb/br/pkg/task/restore.go:534\nmain.runRestoreCommand\n\t/workspace/source/tidb/br/cmd/br/restore.go:58\nmain.newDBRestoreCommand.func1\n\t/workspace/source/tidb/br/cmd/br/restore.go:157\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/workspace/source/tidb/br/cmd/br/main.go:36\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runRestoreCommand\n\t/workspace/source/tidb/br/cmd/br/restore.go:59\nmain.newDBRestoreCommand.func1\n\t/workspace/source/tidb/br/cmd/br/restore.go:157\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/workspace/source/tidb/br/cmd/br/main.go:36\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2025/02/22 00:42:42.627 +00:00] [ERROR] [main.go:38] ["br failed"] [error="context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled"] [errorVerbose="the following errors occurred:\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\n - context canceled\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tidb/br/pkg/task.runRestore\n\t/workspace/source/tidb/br/pkg/task/restore.go:849\ngithub.com/pingcap/tidb/br/pkg/task.RunRestore\n\t/workspace/source/tidb/br/pkg/task/restore.go:534\nmain.runRestoreCommand\n\t/workspace/source/tidb/br/cmd/br/restore.go:58\nmain.newDBRestoreCommand.func1\n\t/workspace/source/tidb/br/cmd/br/restore.go:157\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/workspace/source/tidb/br/cmd/br/main.go:36\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/workspace/source/tidb/br/cmd/br/main.go:38\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]`

### 4. What is your TiDB version? (Required)
./br -V
Release Version: v6.5.12
Git Commit Hash: 05763808fdd7e60ac9609073ee86789ab9ed169a
Git Branch: HEAD
Go Version: go1.19.13
UTC Build Time: 2025-02-18 08:16:39
Race Enabled: false

./tidb-server -V
Release Version: v6.5.12
Edition: Community
Git Commit Hash: 05763808fdd7e60ac9609073ee86789ab9ed169a
Git Branch: HEAD
UTC Build Time: 2025-02-18 08:15:13
GoVersion: go1.19.13
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: unistore

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.