Gitlab backup and restore

Hi everyone.

Please help and provide suggestions for better solution.

Inventory

1. Sameersbn's docker-compose for both gitlabs
2. Both gitlabs are running on 2 different cloud servers (private network connection between both)
3. Production gitlab containers
4. Staging gitlab containers (copy of production gitlab)

Current scenario

We are copying all disk data (gitlab containers volume data) generated by gitlab from production gitlab to staging gitlab. Frequency is everyday. Copying using incremental method.

Copy using rsync cli.

Recreate staging gitlab containers to apply new changes in the data once copy is completed.

All above are done using a shell script with cronjob.

When users are trying to create a project in staging gitlab, they get error saying that There is already a repository with that name on disk. And actually there is no such project used to exist or exist that time in staging gitlab or in production gitlab. But still same issue.

This type of issues automatically get fixed after few minutes/hours.

I believe it has something to do with sidekiq queuing service. Maybe it does not process queue in expected time. Or possibly the queuing service got stuck after staging containers recreate/restart.

Please help in this matter. All constructive suggestions are welcome.
I am thinking of two ways here. Either it should be taken care at Linux OS level with rsync/alternative tool or it should be taken care at gitlab provided inhouse method (rake task).

Basically I am looking for a solution with mostly incremental way, which should done in about <=30 mins. First time it is okay for long time. But further onwards, it should be incremental.

I have share more than enough information here. If any helping mind requires more info, do let me know. Happy to clear doubts.

Appreciate responses in advance.

Screenshot attached

Start by reviewing the docker-compose setup, the shell script, and the cronjob that use rsync to copy production volume data and recreate staging containers. Check how the reported repository-name error relates to the staging restart and the suspected Sidekiq queue or GitLab rake task. Done means an agreed incremental backup and restore process that completes within the stated 30-minute target without producing the error.