[crrf] br log restore fail for "failed to read s3 file...not authorized to perform: s3:GetObject on resource..."
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1. create cluster and enable crrf
2. inject all tikv down
3. pitr restore from the stopped checkpoint
### 2. What did you expect to see? (Required)
restore success
### 3. What did you see instead (Required)
restore fail
```
2026-06-15 12:54:54 | Error: failed to collect from files: failed during reading file v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta: failed to read s3 file, file info: input.bucket='dbaas-dr-staging-us-east-1-1369847559694037350-487825215408', input.key='cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta': operation error S3: GetObject, https response error StatusCode: 403, RequestID: Q07XHFAKFDXJFGM5, HostID: zVKyY7ThJw3mxweAH+8IHA0gF7GoZ/Iyaz6adOrgkW4zSmVpFYM2ik+/alLT9CXW10f0SDyOA5APe/5Fgne+m1TvsfpAhwP3, api error AccessDenied: User: arn:aws:sts::487825215408:assumed-role/ng-role-staging-1369847559694037350-100535366210699981-d53afe4d/i-05cea8328f660a7f4 is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::dbaas-dr-staging-us-east-1-1369847559694037350-487825215408/cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta" because no identity-based policy allows the s3:GetObject action
-- | --
| | 2026-06-15 12:54:54 | [2026/06/15 04:54:54.456 +00:00] [ERROR] [main.go:43] ["br failed"] [error="failed to collect from files: failed during reading file v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta: failed to read s3 file, file info: input.bucket='dbaas-dr-staging-us-east-1-1369847559694037350-487825215408', input.key='cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta': operation error S3: GetObject, https response error StatusCode: 403, RequestID: Q07XHFAKFDXJFGM5, HostID: zVKyY7ThJw3mxweAH+8IHA0gF7GoZ/Iyaz6adOrgkW4zSmVpFYM2ik+/alLT9CXW10f0SDyOA5APe/5Fgne+m1TvsfpAhwP3, api error AccessDenied: User: arn:aws:sts::487825215408:assumed-role/ng-role-staging-1369847559694037350-100535366210699981-d53afe4d/i-05cea8328f660a7f4 is not authorized to perform: s3:GetObject on resource: \"arn:aws:s3:::dbaas-dr-staging-us-east-1-1369847559694037350-487825215408/cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta\" because no identity-based policy allows the s3:GetObject action"] [errorVerbose="operation error S3: GetObject, https response error StatusCode: 403, RequestID: Q07XHFAKFDXJFGM5, HostID: zVKyY7ThJw3mxweAH+8IHA0gF7GoZ/Iyaz6adOrgkW4zSmVpFYM2ik+/alLT9CXW10f0SDyOA5APe/5Fgne+m1TvsfpAhwP3, api error AccessDenied: User: arn:aws:sts::487825215408:assumed-role/ng-role-staging-1369847559694037350-100535366210699981-d53afe4d/i-05cea8328f660a7f4 is not authorized to perform: s3:GetObject on resource: \"arn:aws:s3:::dbaas-dr-staging-us-east-1-1369847559694037350-487825215408/cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta\" because no identity-based policy allows the s3:GetObject action\nfailed to read s3 file, file info: input.bucket='dbaas-dr-staging-us-east-1-1369847559694037350-487825215408', input.key='cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta'\ngithub.com/pingcap/tidb/br/pkg/storage.(*S3Storage).doReadFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:791\ngithub.com/pingcap/tidb/br/pkg/storage.(*S3Storage).ReadFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:763\ngithub.com/pingcap/tidb/br/pkg/restore/log_client.(*LogFileManager).createMetaIterOver.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/log_client/log_file_manager.go:264\ngithub.com/pingcap/tidb/br/pkg/utils/iter.(*chunkMapping[...]).start.func1.2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/iter/combinator_types.go:94\ngithub.com/pingcap/tidb/pkg/util.(*WorkerPool).Apply.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/worker_pool.go:63\nruntime.goexit\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.10.linux-arm64/src/runtime/asm_arm64.s:1268\nfailed during reading file v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta\nfailed to collect from files"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:43\nruntime.main\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.10.linux-arm64/src/runtime/proc.go:285"]
| | 2026-06-15 12:54:54 | [2026/06/15 04:54:54.456 +00:00] [ERROR] [restore.go:77] ["failed to restore"] [error="failed to collect from files: failed during reading file v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta: failed to read s3 file, file info: input.bucket='dbaas-dr-staging-us-east-1-1369847559694037350-487825215408', input.key='cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta': operation error S3: GetObject, https response error StatusCode: 403, RequestID: Q07XHFAKFDXJFGM5, HostID: zVKyY7ThJw3mxweAH+8IHA0gF7GoZ/Iyaz6adOrgkW4zSmVpFYM2ik+/alLT9CXW10f0SDyOA5APe/5Fgne+m1TvsfpAhwP3, api error AccessDenied: User: arn:aws:sts::487825215408:assumed-role/ng-role-staging-1369847559694037350-100535366210699981-d53afe4d/i-05cea8328f660a7f4 is not authorized to perform: s3:GetObject on resource: \"arn:aws:s3:::dbaas-dr-staging-us-east-1-1369847559694037350-487825215408/cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta\" because no identity-based policy allows the s3:GetObject action"] [errorVerbose="operation error S3: GetObject, https response error StatusCode: 403, RequestID: Q07XHFAKFDXJFGM5, HostID: zVKyY7ThJw3mxweAH+8IHA0gF7GoZ/Iyaz6adOrgkW4zSmVpFYM2ik+/alLT9CXW10f0SDyOA5APe/5Fgne+m1TvsfpAhwP3, api error AccessDenied: User: arn:aws:sts::487825215408:assumed-role/ng-role-staging-1369847559694037350-100535366210699981-d53afe4d/i-05cea8328f660a7f4 is not authorized to perform: s3:GetObject on resource: \"arn:aws:s3:::dbaas-dr-staging-us-east-1-1369847559694037350-487825215408/cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta\" because no identity-based policy allows the s3:GetObject action\nfailed to read s3 file, file info: input.bucket='dbaas-dr-staging-us-east-1-1369847559694037350-487825215408', input.key='cluster-10309103528047821478-sr(us-west-2)-dr(us-east-1)/log-781951/v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta'\ngithub.com/pingcap/tidb/br/pkg/storage.(*S3Storage).doReadFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:791\ngithub.com/pingcap/tidb/br/pkg/storage.(*S3Storage).ReadFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:763\ngithub.com/pingcap/tidb/br/pkg/restore/log_client.(*LogFileManager).createMetaIterOver.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/log_client/log_file_manager.go:264\ngithub.com/pingcap/tidb/br/pkg/utils/iter.(*chunkMapping[...]).start.func1.2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/iter/combinator_types.go:94\ngithub.com/pingcap/tidb/pkg/util.(*WorkerPool).Apply.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/worker_pool.go:63\nruntime.goexit\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.10.linux-arm64/src/runtime/asm_arm64.s:1268\nfailed during reading file v1/backupmeta/067B253BF754004500000000000003EC-d067B253AB7CC004Al067B253ABA280026u067B253BF7540085.meta\nfailed to collect from files"] [stack="main.runRestoreCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/restore.go:77\nmain.newStreamRestoreCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/restore.go:251\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:42\nruntime.main\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.10.linux-arm64/src/runtime/proc.go:285"]
| | 2026-06-15 12:54:54 | error: cluster tidb10053536621069998139/restore-2314305-37830a76, wait pipe message failed, errMsg [2026/06/15 04:54:54.419 +00:00] [ERROR] [checkpoint.go:421] ["send the error"] [category=checkpoint] [error="context canceled"] [stack="github.com/pingcap/tidb/br/pkg/checkpoint.(*CheckpointRunner[...]).sendError\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/checkpoint/checkpoint.go:421\ngithub.com/pingcap/tidb/br/pkg/checkpoint.(*CheckpointRunner[...]).startCheckpointMainLoop.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/checkpoint/checkpoint.go:468"]
| | 2026-06-15 12:54:54 | , err: exit status 1
```
### 4. What is your TiDB version? (Required)
[v8.5.6-tidb-advancer-10586-dev](http://gcr.io/pingcap-public/dbaas/tidb:v8.5.6-tidb-advancer-10586-dev)
Contributor guide
Assessment
This issue has not been assessed yet.