Backup cleanup taks fail when using azure storage account and Kubernetes

Bug Report

What version of Kubernetes are you using?

v1.30.4

What version of TiDB Operator are you using?

1.6.0

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

Not aplicable

What's the status of the TiDB cluster pods?

All running fine

What did you do?

Create a TiDB AKS cluster in Microsoft Azure, create a BackupSchedule yaml file.

set spec.backupTemplate.cleanPolicy field to -> Delete.

When the AKS cluster performs the cleanup JOB, the job fails, looking at the storage account, i can see that all files are deleted, but all the folders remain (empty), it causes the cleanup JOB to report fail state.

What did you expect to see?

The cleanup job finish succesfully.

What did you see instead?

The cleanup job fails.

It seems related to the rclone config for azure blob storage, please see: https://github.com/rclone/rclone/issues/7047

The logs from cleanup task container:

Create rclone.conf file.
/tidb-backup-manager clean --namespace=tidb-cluster-01-backup --backupName=schedule-backup-tidb-cluster-01-2024-07-17t06-00-00
Sleeping for 10 seconds before clean...
I0725 06:15:10.839362 9 clean.go:69] start to clean backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00
I0725 06:15:10.840007 9 clean.go:170] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 1, start to clean backup with opt: {PageSize:10000 RetryCount:5 BackoffEnabled:false BatchDeleteOption:{DisableBatchConcurrency:false BatchConcurrency:10 RoutineConcurrency:100} SnapshotsDeleteRatio:0}
I0725 06:15:10.927034 9 clean.go:189] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 1-1, try to delete 4 objects
E0725 06:15:11.002782 9 clean.go:203] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 1-1, delete 4 objects failed
I0725 06:15:11.002816 9 clean.go:223] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 1, clean backup finished, total:4 deleted:0 failed:4
E0725 06:15:11.002825 9 clean.go:163] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 1, failed to clean backup: some objects failed to be deleted
I0725 06:15:11.002830 9 clean.go:170] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 2, start to clean backup with opt: {PageSize:10000 RetryCount:5 BackoffEnabled:false BatchDeleteOption:{DisableBatchConcurrency:false BatchConcurrency:10 RoutineConcurrency:100} SnapshotsDeleteRatio:0}
I0725 06:15:11.047452 9 clean.go:189] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 2-1, try to delete 4 objects
E0725 06:15:11.109367 9 clean.go:203] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 2-1, delete 4 objects failed
I0725 06:15:11.109402 9 clean.go:223] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 2, clean backup finished, total:4 deleted:0 failed:4
E0725 06:15:11.109411 9 clean.go:163] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 2, failed to clean backup: some objects failed to be deleted
I0725 06:15:11.109436 9 clean.go:170] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 3, start to clean backup with opt: {PageSize:10000 RetryCount:5 BackoffEnabled:false BatchDeleteOption:{DisableBatchConcurrency:false BatchConcurrency:10 RoutineConcurrency:100} SnapshotsDeleteRatio:0}
I0725 06:15:11.156129 9 clean.go:189] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 3-1, try to delete 4 objects
E0725 06:15:11.192233 9 clean.go:203] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 3-1, delete 4 objects failed
I0725 06:15:11.192262 9 clean.go:223] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 3, clean backup finished, total:4 deleted:0 failed:4
E0725 06:15:11.192270 9 clean.go:163] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 3, failed to clean backup: some objects failed to be deleted
I0725 06:15:11.192275 9 clean.go:170] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 4, start to clean backup with opt: {PageSize:10000 RetryCount:5 BackoffEnabled:false BatchDeleteOption:{DisableBatchConcurrency:false BatchConcurrency:10 RoutineConcurrency:100} SnapshotsDeleteRatio:0}
I0725 06:15:11.223305 9 clean.go:189] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 4-1, try to delete 4 objects
E0725 06:15:11.262858 9 clean.go:203] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 4-1, delete 4 objects failed
I0725 06:15:11.262905 9 clean.go:223] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 4, clean backup finished, total:4 deleted:0 failed:4
E0725 06:15:11.262913 9 clean.go:163] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 4, failed to clean backup: some objects failed to be deleted
I0725 06:15:11.262918 9 clean.go:170] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 5, start to clean backup with opt: {PageSize:10000 RetryCount:5 BackoffEnabled:false BatchDeleteOption:{DisableBatchConcurrency:false BatchConcurrency:10 RoutineConcurrency:100} SnapshotsDeleteRatio:0}
I0725 06:15:11.303037 9 clean.go:189] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 5-1, try to delete 4 objects
E0725 06:15:11.333956 9 clean.go:203] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 5-1, delete 4 objects failed
I0725 06:15:11.333988 9 clean.go:223] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 5, clean backup finished, total:4 deleted:0 failed:4
E0725 06:15:11.333996 9 clean.go:163] For backup tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 clean 5, failed to clean backup: some objects failed to be deleted
E0725 06:15:11.334005 9 manager.go:102] clean cluster tidb-cluster-01-backup/schedule-backup-tidb-cluster-01-2024-07-17t06-00-00 backup azure://tidb-backups/full-backup/tidb-cluster-01-pd.tidb-cluster-01-2379-2024-07-17t06-00-00/ failed, err: some objects failed to be deleted
error: some objects failed to be deleted
Sleeping for 10 seconds before exit...

Start with the cleanup flow shown in clean.go and manager.go, using the /tidb-backup-manager clean command and the Azure storage configuration described in the report. Reproduce the BackupSchedule cleanup with cleanPolicy set to Delete and trace why object deletion is reported as failed despite files being removed; done means the cleanup job completes successfully without failed deletions.