influxdb v1.8.10 Legacy backup failed

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

use:
/influxd backup -host 127.0.0.1:8088 -database ${db_name} ...
to backup all data in influxdb

__Expected behavior:__
successfully backed up all data(all rp)

__Actual behavior:__
backup process failed when trying to backup one of the rp.
this failure is happens occasionally. We are doing daily back up, what is down below is the frequency of back up failure:
Line 76819: 2022/07/03 01:28:26 backing up db=db rp=rp_8 shard=1684292 to /backup_tmp/db/db.rp_8.1684292.00 since 0001-01-01T00:00:00Z failed.
Line 77054: 2022/07/04 01:29:41 backing up db=db rp=rp_6 shard=1687364 to /backup_tmp/db/db.rp_6.1687364.00 since 0001-01-01T00:00:00Z failed.
Line 78272: 2022/07/06 01:28:18 backing up db=db rp=rp_128 shard=1693571 to /backup_tmp/db/db.rp_128.1693571.00 since 0001-01-01T00:00:00Z failed.
Line 78638: 2022/07/07 01:28:33 backing up db=db rp=rp_22 shard=1696602 to /backup_tmp/db/db.rp_22.1696602.00 since 0001-01-01T00:00:00Z failed.
Line 78851: 2022/07/08 01:28:46 backing up db=db rp=rp_3 shard=1699658 to /backup_tmp/db/db.rp_3.1699658.00 since 0001-01-01T00:00:00Z failed.
Line 79185: 2022/07/09 01:28:20 backing up db=db rp=rp_18 shard=1702746 to /backup_tmp/db/db.rp_18.1702746.00 since 0001-01-01T00:00:00Z failed.
Line 79708: 2022/07/10 01:34:04 backing up db=db rp=rp_44 shard=1705846 to /backup_tmp/db/db.rp_44.1705846.00 since 0001-01-01T00:00:00Z failed.
Line 80122: 2022/07/11 01:28:39 backing up db=db rp=rp_28 shard=1708903 to /backup_tmp/db/db.rp_28.1708903.00 since 0001-01-01T00:00:00Z failed.
Line 80704: 2022/07/12 01:32:24 backing up db=db rp=rp_50 shard=1712001 to /backup_tmp/db/db.rp_50.1712001.00 since 0001-01-01T00:00:00Z failed.
Line 81128: 2022/07/13 01:28:46 backing up db=db rp=rp_29 shard=1715053 to /backup_tmp/db/db.rp_29.1715053.00 since 0001-01-01T00:00:00Z failed.
Line 84520: 2022/07/20 01:19:33 backing up db=db rp=rp_26 shard=1736569 to /backup_tmp/db/db.rp_26.1736569.00 since 0001-01-01T00:00:00Z failed.
Line 86376: 2022/07/22 01:18:38 backing up db=db rp=rp_148 shard=1742774 to /backup_tmp/db/db.rp_148.1742774.00 since 0001-01-01T00:00:00Z failed.
Line 88062: 2022/07/25 01:18:56 backing up db=db rp=rp_128 shard=1751975 to /backup_tmp/db/db.rp_128.1751975.00 since 0001-01-01T00:00:00Z failed.
we can see from this log, sometimes it keeps failing from 2022/07/06 to 2022/07/13. But is works from 2022/07/13 to 2022/07/20. This is why I said this problem happens occasionally.
There is no other error log or info about this back up failure. The only log we have is shown below: backing up a RP and it fails.

__Environment info:__

influxdb v1.8.10
linux 4.19.36 aarch64

__Logs:__
2022/07/29 23:44:36 backing up db=db rp=rp_37 shard=4741 to /backup_tmp/db/pmdb.rp_37.04741.00 since 0001-01-01T00:00:00Z
2022/07/29 23:44:37 backing up db=db rp=rp_38 shard=4742 to /backup_tmp/db/pmdb.rp_38.04742.00 since 0001-01-01T00:00:00Z
2022/07/29 23:44:41 backing up db=db rp=rp_39 shard=4743 to /backup_tmp/db/pmdb.rp_39.04743.00 since 0001-01-01T00:00:00Z
2022/07/29 23:44:43 backing up db=db rp=rp_40 shard=4744 to/backup_tmp/db/pmdb.rp_40.04744.00 since 0001-01-01T00:00:00Z
2022/07/29 23:44:45 backing up db=db rp=rp_41 shard=4745 to /backup_tmp/db/pmdb.rp_41.04745.00 since 0001-01-01T00:00:00Z
2022/07/29 23:44:46 backing up db=db rp=rp_42 shard=4746 to /backup_tmp/db/pmdb.rp_42.04746.00 since 0001-01-01T00:00:00Z failed.

Can someone please help me about this problem. I am so confused right now about what is going on or what will probably cause this backup failure? I am gussing that it might be because influxdb is deleting/dropping retention policy when this backup is processing, which will cause the rp can not be found?

Start by reproducing the failure with the `/influxd backup -host 127.0.0.1:8088 -database ${db_name} ...` command and compare the listed retention-policy failures with the backup logs. Done means identifying why individual shards fail intermittently and confirming that all retention policies are backed up successfully.

Written by the indexing model from the issue text.