cloudnative-pg / cloudnative-pg/klio
Retention policies are not atomic across a backup's snapshots
- Dominant language
- Go
- Stars
- 26
- Forks
- 5
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 113
Description
Retention policies do not see a backup atomically. Each backup is composed of at least three Kopia snapshots, with additional snapshots if there are additional tablespaces. This has the following problems:
1. If a backup fails between snapshots, the previous ones are valid, and will be used to evaluate retention policies. They can trigger deletion of snapshots that are still required by backups that must be retained.
2. They can get different retention labels if they are in different time boundaries (hourly, daily, ...), and this will be extremely hard to detect.
I think we should ditch Kopia retention policies, and have our own retention policy manager, that given a set of retention policies and a catalog of completed backups can identify which ones are outside of retention and then remove them and the WALs that fall outside of retention. Orphan snapshots left by failed backups could be removed if a newer completed backup exists.
Retention policies configuration should be defined alongside the metadata. We can probably keep using the current `klio retention` commands to set them, adding a `klio retention apply` to force the evaluation of the retention policy.
Ditching Kopia retention policies could allow us to choose a simpler retention policy model, like the one in https://cloudnative-pg.io/plugin-barman-cloud/docs/retention/, or even support multiple models.
Contributor guide
Research direction
Start with the existing `klio retention` commands and the backup metadata and catalog of completed backups described in the issue. Define how retention evaluation treats a backup as a unit, including its snapshots and WALs, and how failed backups are handled. Done means retention can be applied consistently, with an optional `klio retention apply` command and no deletion of snapshots still needed by retained backups.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100