bcgov / bcgov/backup-container

Add retry to the backup process

Open
#60 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue help wanted
Dominant language
Shell
Stars
44
Forks
65
PR merge metrics
No merged PRs in 30d

Description

Occasionally a backup can run into an issue and fail on the first attempt. If attempted again a sort time later it will succeed.

For the sake of robustness add a configurable retry and delay to the backup process so backups are retried if they fail the first time.

Example backup failure:
```
Backing up 'wallet-indy-cat:5432/agent_indy_cat_wallet' to '/backups/daily/2020-08-22/wallet-indy-cat-agent_indy_cat_wallet_2020-08-22_01-05-02.sql.gz.in_progress' ...
--
  | pg_dump: Dumping the contents of table "items" failed: PQgetCopyData() failed.
  | pg_dump: Error message from server: server closed the connection unexpectedly
  | This probably means the server terminated abnormally
  | before or while processing the request.
  | pg_dump: The command was: COPY public.items (id, type, name, value, key) TO stdout;
  | removed '/backups/daily/2020-08-22/wallet-indy-cat-agent_indy_cat_wallet_2020-08-22_01-05-02.sql.gz.in_progress'
  | [!!ERROR!!] - Failed to backup wallet-indy-cat:5432/agent_indy_cat_wallet.
  |  
  | Elapsed time: 0h:25m:53s - Status Code: 1
```

The database server did not terminate. When the backup was retried it succeeded.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.