pingcap / pingcap/tidb

Backup: failed due to external storage internal error.

Open
#35,469 0 comments 0 reactions 0 assignees View on GitHub
component/br severity/minor type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
In https://github.com/pingcap/tidb/issues/34350 we mainly retry the error that TiKV return to br. but in some cases, br alse needs handle the error from external storage.

for example:
br will write meta/lock file to external storage.
And br should tolerate the 5xx error of external storage. and give some fundamental retry.

### 2. What did you expect to see? (Required)
br can retry the 5xx error from external storage.

### 3. What did you see instead (Required)
failed
```
2022/06/17 02:00:48.571 +00:00] [ERROR] [backup.go:41] ["failed to backup"] [error="googleapi: Error 503: We encountered an internal error. Please try again., backendError"] [errorVerbose="googleapi: Error 503: We encountered an internal error. Please try again., backendError\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\ngithub.com/pingcap/br/pkg/task.RunBackup\n\tgithub.com/pingcap/br@/pkg/task/backup.go:245\nmain.runBackupCommand\n\tgithub.com/pingcap/br@/cmd/br/backup.go:40\nmain.newFullBackupCommand.func1\n\tgithub.com/pingcap/br@/cmd/br/backup.go:109\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\tgithub.com/pingcap/br@/cmd/br/main.go:56\nruntime.main\n\truntime/proc.go:203\nruntime.goexit\n\truntime/asm_amd64.s:1357"] [stack="main.runBackupCommand\n\tgithub.com/pingcap/br@/cmd/br/backup.go:41\nmain.newFullBackupCommand.func1\n\tgithub.com/pingcap/br@/cmd/br/backup.go:109\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\tgithub.com/pingcap/br@/cmd/br/main.go:56\nruntime.main\n\truntime/proc.go:203"]
```

### 4. What is your TiDB version? (Required)

master

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.