Dokploy / Dokploy/dokploy

Backup retention deletes other backups' files when two backups share a database service

Open
#4,914 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
37.4k
Forks
3k
Avg merge
1d 3h
Merged PRs (30d)
73

Description

To Reproduce

Two backups configured on the same database service (a single Postgres instance hosting two databases) silently delete each other's files in S3.

  1. Create a Postgres service that hosts two databases, e.g. app and analytics.
  2. Create backup A: database app, schedule 0 0 * * * (daily), Keep latest = 7, destination my-bucket, prefix left as the default /.
  3. Create backup B: database analytics, schedule 0 0 * * 0 (weekly), Keep latest = 2, same destination, same default prefix /.
  4. Let both run for a week.
  5. List the bucket: rclone lsf :s3:my-bucket/<postgres-appName>/

Only 2 files remain, not 9. Every Sunday, backup B's retention deletes 5 of backup A's daily dumps.

Current vs. Expected behavior

Current. Both backups write to the same key prefix and the filename carries no identity, so retention cannot tell the two apart:

  • packages/server/src/utils/backups/postgres.ts#L34-L35 — the object key is ${appName}/${normalizeS3Path(prefix)}${getBackupTimestamp()}.sql.gz. appName is the service name, not the database name, and the filename is only a timestamp. Two backups on the same service with the same prefix therefore land in the same folder with indistinguishable names.
  • packages/server/src/utils/backups/index.ts#L127-L159 (keepLatestNBackups) — lists the whole folder (--include "*.{sql.gz,bson.gz}"), sorts, and deletes everything past that backup's own keepLatestCount:
    rclone lsf … --include "*.{sql.gz,bson.gz}" :s3:bucket/<appName>/<prefix>
      | sort -r | tail -n +$((keepLatestCount+1)) | xargs -I{} rclone delete … {}
    
    The list is not scoped to the backup that owns it, so the smallest keepLatestCount among the backups sharing the folder governs all of them.

The failure is silent: both backups report ✅, the UI shows "Keep latest 7", and the retention that actually applies is 2.

Expected. A backup's retention should only ever consider files it produced. keepLatestCount = 7 should keep 7 copies of that database.

Observed in production. A 9.4 GB database backed up daily with keep=7, sharing a service with a small database backed up weekly with keep=2: the bucket held 1 dump of the main database instead of 7 — i.e. the real RPO was ~1 day and one bad restore away from nothing, while the UI claimed a week of history.

Suggested fix

Any of these would close it (roughly in order of preference):

  1. Put the database name in the object key — ${database}-${timestamp}.sql.gz — and scope the retention --include to ${database}-*. This also makes the bucket readable by a human, which it currently isn't.
  2. Default prefix to the database name instead of / when a service already has a backup for a different database.
  3. At minimum, reject/warn in the UI when a new backup would share (service, destination, prefix) with an existing one.

The workaround for existing users is to set a distinct prefix per backup, but nothing hints that it is required, and files already written to the shared folder then fall out of every retention and are never cleaned up.

Provide environment information
Operating System:
  OS: Ubuntu 24.04.4 LTS
  Arch: x86_64
Docker: 29.5.3 (swarm active)
Dokploy version: 0.29.10
  (code path verified unchanged in v0.29.13 — links above point at that tag)
VPS Provider: self-hosted VPS
What applications/services: Postgres 18 database service with two databases, backups to Cloudflare R2
Which area(s) are affected?

Databases, Backups

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read packages/server/src/utils/backups/postgres.ts around lines 34-35 and packages/server/src/utils/backups/index.ts around lines 127-159; trace how backup object keys are formed and how keepLatestNBackups selects files. Compare the available fixes with the expected behavior, then verify that retention only considers files produced by the individual database backup and preserves each configured count.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, typescript
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.