Backup to Alibaba OSS error（dumpling）：Rclone S3 excess default ACL exists

Deployment environment

• kubernetes version: v1.24.2
• tidb operator version: v1.3.9

bk

cat <<-EOF | kubectl apply -f -
---
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
  name: gpsp
  namespace: tidb-cluster
spec:
  from:
    host: gpsp-tidb
    port: 4000
    user: root
    secretName: gpsp-tidb-backup-secret
  s3:
    provider: alibaba
    region: oss-cn-zhangjiakou
    bucket: gpsp
    prefix: tidb-backup
    endpoint: https://gpsp.oss-cn-zhangjiakou.aliyuncs.com
    secretName: oss-secret
  dumpling:
    options:
      - --threads=16
      - --rows=10000
  storageClassName: longhorn
  storageSize: 100Gi
EOF

error logs

Create rclone.conf file.
/tidb-backup-manager export --namespace=tidb-cluster --backupName=gpsp --bucket=gpsp --storageType=s3
I1106 01:58:46.110902       9 export.go:74] start to process backup tidb-cluster/gpsp
I1106 01:58:46.152348       9 backup_status_updater.go:86] Backup: [tidb-cluster/gpsp] updated successfully
I1106 01:58:46.160026       9 manager.go:179] cluster tidb-cluster/gpsp tikv_gc_life_time is 10m0s
I1106 01:58:46.173253       9 manager.go:243] set cluster tidb-cluster/gpsp tikv_gc_life_time to 72h success
E1106 01:58:46.184620       9 backup_status_updater.go:89] Failed to update backup [tidb-cluster/gpsp], error: Operation cannot be fulfilled on backups.pingcap.com "gpsp": the object has been modified; please apply your changes to the latest version and try again
I1106 01:58:46.209279       9 backup_status_updater.go:86] Backup: [tidb-cluster/gpsp] updated successfully
I1106 01:58:46.219249       9 export.go:97] The dump process is ready, command "/dumpling --output=/backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z --host=gpsp-tidb --port=4000 --user=root --password=****** --filter *.* --filter !/^(mysql|test|INFORMATION_SCHEMA|PERFORMANCE_SCHEMA|METRICS_SCHEMA|INSPECTION_SCHEMA)$/.* --threads=16 --rows=10000"
I1106 02:00:19.057815       9 manager.go:284] reset cluster tidb-cluster/gpsp tikv_gc_life_time to 10m0s success
I1106 02:00:19.057853       9 manager.go:299] dump cluster tidb-cluster/gpsp data to /backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z success
I1106 02:00:19.058051       9 manager.go:314] get cluster tidb-cluster/gpsp commitTs 437169979755855873 success
I1106 02:05:14.457187       9 manager.go:329] archive cluster tidb-cluster/gpsp backup data /backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz success
I1106 02:05:14.487832       9 manager.go:345] get cluster tidb-cluster/gpsp archived backup file /backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz size 1120088963 success
I1106 02:05:44.599196       9 export.go:116] cluster tidb-cluster/gpsp, rclone copy data from /backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz to s3:gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz.tmp, log: 2022/11/06 02:05:18 INFO  : S3 bucket gpsp path tidb-backup: Bucket "gpsp" created with ACL "private"
2022/11/06 02:05:44 INFO  : backup-2022-11-05T17:58:46Z.tgz: Copied (new)
2022/11/06 02:05:44 INFO  :
Transferred:   	    1.043G / 1.043 GBytes, 100%, 40.310 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        26.4s

I1106 02:05:44.599258       9 export.go:117] upload cluster tidb-cluster/gpsp backup data to s3:gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz.tmp successfully, now move it to permanent URL s3:gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz
E1106 02:05:45.470609       9 manager.go:353] backup cluster tidb-cluster/gpsp data to s3 failed, err: cluster tidb-cluster/gpsp, execute rclone moveto command failed, output: 2022/11/06 02:05:45 INFO  : S3 bucket gpsp path tidb-backup: Bucket "gpsp" created with ACL "private"
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : Attempt 1/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : Attempt 2/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 ERROR : Attempt 3/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 Failed to moveto: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC49363107C581, host id:
, err: exit status 1
E1106 02:05:45.482902       9 backup_status_updater.go:89] Failed to update backup [tidb-cluster/gpsp], error: Operation cannot be fulfilled on backups.pingcap.com "gpsp": the object has been modified; please apply your changes to the latest version and try again
I1106 02:05:45.516995       9 backup_status_updater.go:86] Backup: [tidb-cluster/gpsp] updated successfully
error: cluster tidb-cluster/gpsp, execute rclone moveto command failed, output: 2022/11/06 02:05:45 INFO  : S3 bucket gpsp path tidb-backup: Bucket "gpsp" created with ACL "private"
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : Attempt 1/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : Attempt 2/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 ERROR : Attempt 3/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 Failed to moveto: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 6366A5F942FC49363107C581, host id:
, err: exit status 1

Start by tracing the backup flow through export.go and manager.go, focusing on the rclone copy and moveto commands shown in the logs. Reproduce the Alibaba OSS configuration and inspect how the temporary object path and ACL are handled. Done means the backup moves to its permanent OSS URL without the NoSuchKey failure.