Backup to Alibaba OSS error（br）：Alibaba RAM Provider Retrieve Failed

Deployment environment

• kubernetes version：v1.24.2
• tidb operator version：v1.3.9

bk

cat <<-EOF | kubectl apply -f -
---
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
  name: gpsp
  namespace: tidb-cluster
spec:
  backupType: full
  br:
    cluster: "gpsp"
    clusterNamespace: "tidb-cluster"
    sendCredToTikv: false
  tableFilter:
  - "gpsp_dev.*"
  - "gpsp_prod.*"
  - "mysql.user"
  s3:
    provider: alibaba
    secretName: oss-secret
    region: oss-cn-zhangjiakou
    bucket: gpsp
    prefix: tidb-backup
    endpoint: https://oss-cn-zhangjiakou.aliyuncs.com
  storageSize: 100Gi
EOF

error logs

Defaulted container "backup" out of: backup, br (init)
Create rclone.conf file.
/tidb-backup-manager backup --namespace=tidb-cluster --backupName=gpsp --tikvVersion=v6.1.0
I1106 00:57:17.703151       9 backup.go:72] start to process backup tidb-cluster/gpsp
I1106 00:57:17.738408       9 backup_status_updater.go:86] Backup: [tidb-cluster/gpsp] updated successfully
E1106 00:57:17.748668       9 backup_status_updater.go:89] Failed to update backup [tidb-cluster/gpsp], error: Operation cannot be fulfilled on backups.pingcap.com "gpsp": the object has been modified; please apply your changes to the latest version and try again
I1106 00:57:17.783049       9 backup_status_updater.go:86] Backup: [tidb-cluster/gpsp] updated successfully
I1106 00:57:17.783105       9 backup.go:69] Running br command with args: [backup full --pd=gpsp-pd.tidb-cluster:2379 --send-credentials-to-tikv=false --storage=s3://gpsp/tidb-backup --s3.region=oss-cn-zhangjiakou --s3.provider=alibaba --s3.endpoint=https://oss-cn-zhangjiakou.aliyuncs.com --filter gpsp_dev.* --filter gpsp_prod.* --filter mysql.user]
I1106 00:57:17.849943       9 backup.go:93] [2022/11/06 00:57:17.849 +08:00] [INFO] [info.go:49] ["Welcome to Backup & Restore (BR)"] [release-version=v6.1.0] [git-hash=1a89decdb192cbdce6a7b0020d71128bc964d30f] [git-branch=heads/refs/tags/v6.1.0] [go-version=go1.18.2] [utc-build-time="2022-06-05 05:09:12"] [race-enabled=false]
I1106 00:57:17.849977       9 backup.go:93] [2022/11/06 00:57:17.849 +08:00] [INFO] [common.go:692] [arguments] [__command="br backup full"] [filter="[gpsp_dev.*,gpsp_prod.*,mysql.user]"] [pd="[gpsp-pd.tidb-cluster:2379]"] [s3.endpoint=https://oss-cn-zhangjiakou.aliyuncs.com] [s3.provider=alibaba] [s3.region=oss-cn-zhangjiakou] [send-credentials-to-tikv=false] [storage=s3://gpsp/tidb-backup]
I1106 00:57:17.850122       9 backup.go:93] [2022/11/06 00:57:17.850 +08:00] [INFO] [conn.go:257] ["new mgr"] [pdAddrs=gpsp-pd.tidb-cluster:2379]
I1106 00:57:17.862213       9 backup.go:93] [2022/11/06 00:57:17.862 +08:00] [INFO] [client.go:392] ["[pd] create pd client with endpoints"] [pd-address="[gpsp-pd.tidb-cluster:2379]"]
I1106 00:57:17.871492       9 backup.go:93] [2022/11/06 00:57:17.871 +08:00] [INFO] [base_client.go:332] ["[pd] update member urls"] [old-urls="[http://gpsp-pd.tidb-cluster:2379]"] [new-urls="[http://gpsp-pd-0.gpsp-pd-peer.tidb-cluster.svc:2379,http://gpsp-pd-1.gpsp-pd-peer.tidb-cluster.svc:2379,http://gpsp-pd-2.gpsp-pd-peer.tidb-cluster.svc:2379]"]
I1106 00:57:17.871700       9 backup.go:93] [2022/11/06 00:57:17.871 +08:00] [INFO] [base_client.go:350] ["[pd] switch leader"] [new-leader=http://gpsp-pd-1.gpsp-pd-peer.tidb-cluster.svc:2379] [old-leader=]
I1106 00:57:17.871710       9 backup.go:93] [2022/11/06 00:57:17.871 +08:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=7162231159068286236]
I1106 00:57:17.871921       9 backup.go:93] [2022/11/06 00:57:17.871 +08:00] [INFO] [client.go:687] ["[pd] tso dispatcher created"] [dc-location=global]
I1106 00:57:17.881070       9 backup.go:93] [2022/11/06 00:57:17.880 +08:00] [INFO] [conn.go:232] ["checked alive KV stores"] [aliveStores=3] [totalStores=3]
I1106 00:57:17.881100       9 backup.go:93] [2022/11/06 00:57:17.880 +08:00] [INFO] [client.go:392] ["[pd] create pd client with endpoints"] [pd-address="[gpsp-pd.tidb-cluster:2379]"]
I1106 00:57:17.892962       9 backup.go:93] [2022/11/06 00:57:17.892 +08:00] [INFO] [base_client.go:332] ["[pd] update member urls"] [old-urls="[http://gpsp-pd.tidb-cluster:2379]"] [new-urls="[http://gpsp-pd-0.gpsp-pd-peer.tidb-cluster.svc:2379,http://gpsp-pd-1.gpsp-pd-peer.tidb-cluster.svc:2379,http://gpsp-pd-2.gpsp-pd-peer.tidb-cluster.svc:2379]"]
I1106 00:57:17.893040       9 backup.go:93] [2022/11/06 00:57:17.892 +08:00] [INFO] [base_client.go:350] ["[pd] switch leader"] [new-leader=http://gpsp-pd-1.gpsp-pd-peer.tidb-cluster.svc:2379] [old-leader=]
I1106 00:57:17.893065       9 backup.go:93] [2022/11/06 00:57:17.893 +08:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=7162231159068286236]
I1106 00:57:17.893833       9 backup.go:93] [2022/11/06 00:57:17.893 +08:00] [INFO] [client.go:687] ["[pd] tso dispatcher created"] [dc-location=global]
I1106 00:57:17.901182       9 backup.go:93] [2022/11/06 00:57:17.900 +08:00] [INFO] [tidb.go:74] ["new domain"] [store=tikv-7162231159068286236] ["ddl lease"=1s] ["stats lease"=-1ns] ["index usage sync lease"=0s]
I1106 00:57:17.939447       9 backup.go:93] [2022/11/06 00:57:17.939 +08:00] [INFO] [domain.go:1731] [acquireServerID] [serverID=2235512] ["lease id"=6fcf8443ec93625f]
I1106 00:57:17.994246       9 backup.go:93] [2022/11/06 00:57:17.993 +08:00] [WARN] [info.go:241] ["init TiFlashPlacementManager"] ["pd addrs"="[gpsp-pd-1.gpsp-pd-peer.tidb-cluster.svc:2379,gpsp-pd-2.gpsp-pd-peer.tidb-cluster.svc:2379,gpsp-pd-0.gpsp-pd-peer.tidb-cluster.svc:2379]"]
I1106 00:57:18.175965       9 backup.go:93] [2022/11/06 00:57:18.175 +08:00] [INFO] [domain.go:176] ["full load InfoSchema success"] [currentSchemaVersion=0] [neededSchemaVersion=442] ["start time"=111.383324ms]
I1106 00:57:18.186295       9 backup.go:93] [2022/11/06 00:57:18.186 +08:00] [INFO] [domain.go:437] ["full load and reset schema validator"]
I1106 00:57:18.186320       9 backup.go:93] [2022/11/06 00:57:18.186 +08:00] [INFO] [ddl.go:382] ["[ddl] start DDL"] [ID=8aa45ab1-014d-4e47-9c72-68b082dc6e35] [runWorker=false]
I1106 00:57:18.190451       9 backup.go:93] [2022/11/06 00:57:18.190 +08:00] [INFO] [backup.go:273] ["get new_collations_enabled_on_first_bootstrap config from system table"] [new_collation_enabled=True]
I1106 00:57:18.190478       9 backup.go:93] [2022/11/06 00:57:18.190 +08:00] [INFO] [client.go:95] ["new backup client"]
I1106 00:57:48.192155       9 backup.go:93] [2022/11/06 00:57:48.191 +08:00] [INFO] [client.go:768] ["[pd] stop fetching the pending tso requests due to context canceled"] [dc-location=global]
I1106 00:57:48.192239       9 backup.go:93] [2022/11/06 00:57:48.192 +08:00] [INFO] [client.go:706] ["[pd] exit tso dispatcher"] [dc-location=global]
I1106 00:57:48.193738       9 backup.go:93] [2022/11/06 00:57:48.193 +08:00] [INFO] [client.go:768] ["[pd] stop fetching the pending tso requests due to context canceled"] [dc-location=global]
I1106 00:57:48.193772       9 backup.go:93] [2022/11/06 00:57:48.193 +08:00] [INFO] [client.go:706] ["[pd] exit tso dispatcher"] [dc-location=global]
I1106 00:57:48.194065       9 backup.go:93] [2022/11/06 00:57:48.193 +08:00] [INFO] [collector.go:204] ["units canceled"] [cancel-unit=0]
I1106 00:57:48.194080       9 backup.go:93] [2022/11/06 00:57:48.193 +08:00] [INFO] [collector.go:205] ["Full Backup failed summary"] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0]
I1106 00:57:48.194792       9 backup.go:93] [2022/11/06 00:57:48.193 +08:00] [ERROR] [backup.go:40] ["failed to backup"] [error="Alibaba RAM Provider Retrieve: Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout"] [errorVerbose="Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout\nAlibaba RAM Provider Retrieve\ngithub.com/pingcap/tidb/br/pkg/storage.createOssRamCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:269\ngithub.com/pingcap/tidb/br/pkg/storage.autoNewCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:260\ngithub.com/pingcap/tidb/br/pkg/storage.newS3Storage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:287\ngithub.com/pingcap/tidb/br/pkg/storage.New\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/storage.go:167\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).SetStorage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:176\ngithub.com/pingcap/tidb/br/pkg/task.RunBackup\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup.go:284\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:39\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:108\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:40\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:108\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
I1106 00:57:48.194881       9 backup.go:93] [2022/11/06 00:57:48.194 +08:00] [ERROR] [main.go:59] ["br failed"] [error="Alibaba RAM Provider Retrieve: Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout"] [errorVerbose="Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout\nAlibaba RAM Provider Retrieve\ngithub.com/pingcap/tidb/br/pkg/storage.createOssRamCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:269\ngithub.com/pingcap/tidb/br/pkg/storage.autoNewCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:260\ngithub.com/pingcap/tidb/br/pkg/storage.newS3Storage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:287\ngithub.com/pingcap/tidb/br/pkg/storage.New\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/storage.go:167\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).SetStorage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:176\ngithub.com/pingcap/tidb/br/pkg/task.RunBackup\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup.go:284\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:39\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:108\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:59\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
I1106 00:57:48.201446       9 backup.go:93]
I1106 00:57:48.201532       9 backup.go:100] Error: Alibaba RAM Provider Retrieve: Get "http://100.100.100.200/latest/meta-data/ram/security-credentials/": dial tcp 100.100.100.200:80: i/o timeout
E1106 00:57:48.201795       9 manager.go:293] backup cluster tidb-cluster/gpsp data failed, err: cluster tidb-cluster/gpsp, wait pipe message failed, errMsg [2022/11/06 00:57:48.193 +08:00] [ERROR] [backup.go:40] ["failed to backup"] [error="Alibaba RAM Provider Retrieve: Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout"] [errorVerbose="Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout\nAlibaba RAM Provider Retrieve\ngithub.com/pingcap/tidb/br/pkg/storage.createOssRamCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:269\ngithub.com/pingcap/tidb/br/pkg/storage.autoNewCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:260\ngithub.com/pingcap/tidb/br/pkg/storage.newS3Storage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:287\ngithub.com/pingcap/tidb/br/pkg/storage.New\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/storage.go:167\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).SetStorage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:176\ngithub.com/pingcap/tidb/br/pkg/task.RunBackup\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup.go:284\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:39\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:108\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:40\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:108\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2022/11/06 00:57:48.194 +08:00] [ERROR] [main.go:59] ["br failed"] [error="Alibaba RAM Provider Retrieve: Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout"] [errorVerbose="Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout\nAlibaba RAM Provider Retrieve\ngithub.com/pingcap/tidb/br/pkg/storage.createOssRamCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:269\ngithub.com/pingcap/tidb/br/pkg/storage.autoNewCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:260\ngithub.com/pingcap/tidb/br/pkg/storage.newS3Storage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:287\ngithub.com/pingcap/tidb/br/pkg/storage.New\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/storage.go:167\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).SetStorage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:176\ngithub.com/pingcap/tidb/br/pkg/task.RunBackup\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup.go:284\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:39\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:108\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:59\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
Error: Alibaba RAM Provider Retrieve: Get "http://100.100.100.200/latest/meta-data/ram/security-credentials/": dial tcp 100.100.100.200:80: i/o timeout
, err: exit status 1
E1106 00:57:48.220045       9 backup_status_updater.go:89] Failed to update backup [tidb-cluster/gpsp], error: Operation cannot be fulfilled on backups.pingcap.com "gpsp": the object has been modified; please apply your changes to the latest version and try again
I1106 00:57:48.255659       9 backup_status_updater.go:86] Backup: [tidb-cluster/gpsp] updated successfully
error: cluster tidb-cluster/gpsp, wait pipe message failed, errMsg [2022/11/06 00:57:48.193 +08:00] [ERROR] [backup.go:40] ["failed to backup"] [error="Alibaba RAM Provider Retrieve: Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout"] [errorVerbose="Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout\nAlibaba RAM Provider Retrieve\ngithub.com/pingcap/tidb/br/pkg/storage.createOssRamCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:269\ngithub.com/pingcap/tidb/br/pkg/storage.autoNewCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:260\ngithub.com/pingcap/tidb/br/pkg/storage.newS3Storage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:287\ngithub.com/pingcap/tidb/br/pkg/storage.New\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/storage.go:167\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).SetStorage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:176\ngithub.com/pingcap/tidb/br/pkg/task.RunBackup\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup.go:284\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:39\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:108\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:40\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:108\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2022/11/06 00:57:48.194 +08:00] [ERROR] [main.go:59] ["br failed"] [error="Alibaba RAM Provider Retrieve: Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout"] [errorVerbose="Get \"http://100.100.100.200/latest/meta-data/ram/security-credentials/\": dial tcp 100.100.100.200:80: i/o timeout\nAlibaba RAM Provider Retrieve\ngithub.com/pingcap/tidb/br/pkg/storage.createOssRamCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:269\ngithub.com/pingcap/tidb/br/pkg/storage.autoNewCred\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:260\ngithub.com/pingcap/tidb/br/pkg/storage.newS3Storage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:287\ngithub.com/pingcap/tidb/br/pkg/storage.New\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/storage.go:167\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).SetStorage\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:176\ngithub.com/pingcap/tidb/br/pkg/task.RunBackup\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup.go:284\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:39\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:108\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:59\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
Error: Alibaba RAM Provider Retrieve: Get "http://100.100.100.200/latest/meta-data/ram/security-credentials/": dial tcp 100.100.100.200:80: i/o timeout
, err: exit status 1

Start by tracing the Alibaba credential path in br/pkg/storage/s3.go, especially createOssRamCred at line 269, and reproduce the Backup command with the shown OSS secret and endpoint. Determine why the backup falls back to the Alibaba RAM metadata endpoint and verify that the backup completes using the configured credentials.