bcgov / bcgov/backup-container

Add retry to Verification process

Open
#72 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue help wanted
Dominant language
Shell
Stars
44
Forks
65
PR merge metrics
No merged PRs in 30d

Description

Occasionally a verification can run into an issue and fail on the first attempt. If attempted again a sort time later it will succeed.

For the sake of robustness add a configurable retry and delay to the verification process so verifications are retried if they fail the first time.

Example failure:
```
Verifying backup ...
  |  
  | Settings:
  | - Database: postgres=vc-authn-database/vc_authn
  | - Backup file: /backups/daily/2021-01-28/vc-authn-database-vc_authn_2021-01-28_01-00-00.sql.gz
  |  
  | Already clean ...
  |  
  | waiting for server to start.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
  | The server failed to start within 0h:10m:0s.
```

In this case the local server used to verify the backup did not start in the allotted time (and as you can see it was given plenty of time to startup). This was the first verification of four this server performs daily. The subsequent three verifications competed successfully with the local server starting in around 2 minutes (slower than expected, but still acceptable for the purpose of automated verification).

```

Verifying backup ...
--
  |  
  | Settings:
  | - Database: postgres=vc-authn-wallet/vc_authn_agent_wallet
  | - Backup file: /backups/daily/2021-01-28/vc-authn-wallet-vc_authn_agent_wallet_2021-01-28_01-00-03.sql.gz
  |  
  | Already clean ...
  |  
  | waiting for server to start....................................................................................................
  | The server started in 0h:1m:45s.
  |  
  |  
  | Restoring from backup ...
  | Restoring '/backups/daily/2021-01-28/vc-authn-wallet-vc_authn_agent_wallet_2021-01-28_01-00-03.sql.gz' to '127.0.0.1/vc_authn_agent_wallet' ...
  | waiting for server to shut down.... done
  | server stopped
  | Cleaning up ...
  |  
  | Successfully verified backup: /backups/daily/2021-01-28/vc-authn-wallet-vc_authn_agent_wallet_2021-01-28_01-00-03.sql.gz
  | The restored database contained 4 tables, and is 7783 kB in size.
  |  
  | Elapsed time: 0h:1m:55s - Status Code: 0
  |  
  | Verifying backup ...
  |  
  | Settings:
  | - Database: postgres=wallet-lsbc/agent_lsbc_wallet
  | - Backup file: /backups/daily/2021-01-28/wallet-lsbc-agent_lsbc_wallet_2021-01-28_01-00-07.sql.gz
  |  
  | Already clean ...
  |  
  | waiting for server to start..................................................................................................
  | The server started in 0h:1m:43s.
  |  
  |  
  | Restoring from backup ...
  | Restoring '/backups/daily/2021-01-28/wallet-lsbc-agent_lsbc_wallet_2021-01-28_01-00-07.sql.gz' to '127.0.0.1/agent_lsbc_wallet' ...
  | waiting for server to shut down.... done
  | server stopped
  | Cleaning up ...
  |  
  | Successfully verified backup: /backups/daily/2021-01-28/wallet-lsbc-agent_lsbc_wallet_2021-01-28_01-00-07.sql.gz
  | The restored database contained 4 tables, and is 7783 kB in size.
  |  
  | Elapsed time: 0h:2m:58s - Status Code: 0
  |  
  | Verifying backup ...
  |  
  | Settings:
  | - Database: postgres=wallet-prime/agent_prime_wallet
  | - Backup file: /backups/daily/2021-01-28/wallet-prime-agent_prime_wallet_2021-01-28_01-00-11.sql.gz
  |  
  | Already clean ...
  |  
  | waiting for server to start......................................................................................................................
  | The server started in 0h:2m:4s.
  |  
  |  
  | Restoring from backup ...
  | Restoring '/backups/daily/2021-01-28/wallet-prime-agent_prime_wallet_2021-01-28_01-00-11.sql.gz' to '127.0.0.1/agent_prime_wallet' ...
  | waiting for server to shut down.... done
  | server stopped
  | Cleaning up ...
  |  
  | Successfully verified backup: /backups/daily/2021-01-28/wallet-prime-agent_prime_wallet_2021-01-28_01-00-11.sql.gz
  | The restored database contained 4 tables, and is 7927 kB in size.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.