bcgov / bcgov/backup-container

Backups can fail silently - Failed backup notifications are not being sent

Open
#140 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Shell
Stars
44
Forks
65
PR merge metrics
No merged PRs in 30d

Description

The latest version of the backup-container appears to be failing to send failed backup notifications.

We encountered an issue where the backup volume filled up and was causing backups to fail.

```
Backing up 'wallet-idim/agent_idim_wallet' to '/backups/daily/2025-02-03/wallet-idim-agent_idim_wallet_2025-02-03_01-00-00.sql.gz.in_progress' ...

gzip: stdout: No space left on device
removed '/backups/daily/2025-02-03/wallet-idim-agent_idim_wallet_2025-02-03_01-00-00.sql.gz.in_progress'
[!!ERROR!!] - Failed to backup postgres=wallet-idim/agent_idim_wallet.

Elapsed time: 0h:0m:56s - Status Code: 141
```

The notifications for these backup failures was never sent, however the subsequent verification of a backup that was corrupted by this issue did cause a failed verification notification to be sent.

This means backups can fail silently and if there is no subsequent failed verification notification the issue can go unnoticed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.