Scheduled backups run twice: upstream CronJob and Percona in-process scheduler both fire for the same schedule, racing for the pgBackRest backup lock

Report

In operator 3.0.0, every backup schedule defined in spec.backups.pgbackrest.repos[].schedules is executed by two independent mechanisms simultaneously:

1. The upstream (CrunchyData) reconciler creates a Kubernetes CronJob (<cluster>-repo1-<type>) whose pod runs pgbackrest backup directly (internal/controller/postgrescluster/pgbackrest.go, ~L3184).
2. The Percona layer registers an in-process cron for the same schedule (percona/controller/pgcluster/schedule.go — reconcileScheduledBackup → createScheduledBackup) which creates a PerconaPGBackup CR (GenerateName: <cronjob-name>-, no annotations) ~30s later.

The PerconaPGBackup controller never "adopts" the CronJob's job: findBackupJob() (percona/controller/pgbackup/controller.go, ~L702-739) only matches manual-type jobs annotated with the CR's name. So reconciling the scheduler-created CR calls startBackup() and launches a second, manual-type backup job (<cluster>-backup-xxxx) for the same slot. Both pgbackrest processes race for the stanza's backup lock on the repo host:

ERROR: [050]: unable to acquire lock on file '/tmp/pgbackrest/db-backup-1.lock': Resource temporarily unavailable
HINT: is another pgBackRest process running?

Additionally, reconcileBackupJob() (percona/controller/pgcluster/backup.go, ~L187) creates a third object — a tracking PerconaPGBackup CR for the CronJob's job (annotated with pgv2.percona.com/pgbackrest-backup-job-name) — so two CRs accumulate per schedule slot.

More about the problem

The duplicate execution cascades into a self-perpetuating failure loop:

• Whichever process loses the lock race fails with exit 50; scheduled CronJob jobs intermittently end BackoffLimitExceeded (slots missing their backup, retries succeeding at odd times).
• When the duplicate manual job's pods exhaust retries and the job is removed, its CR remains in Starting forever — the BackupStarting branch requeues "Waiting for backup to start" every 5s with no timeout (percona/controller/pgbackup/controller.go, ~L240).
• The stuck CR holds the controller's lease (handleLease), freezing reconciliation of all pg-backup CRs for the cluster. In our production cluster, 936 unreconciled CRs accumulated over ~6 weeks behind a CR stuck in Starting.
• Deleting the stuck CR unfreezes the queue; the backlog then fires manual backups at arbitrary times, colliding with the scheduled CronJobs again and producing a new stuck CR. We reproduced and recorded this live.

Operator log while stuck (repeats every 5s, indefinitely):

INFO Waiting for backup to start {"controller": "perconapgbackup", "controllerKind": "PerconaPGBackup", "name": "postgres-ash-vega-prod-repo1-incr-7nskz", ...}

ps on the repo host during a slot — the CR-driven backup (note the --annotation) holding the lock while the scheduled CronJob's pods crash-loop with exit 50:

pgbackrest backup --stanza=db --repo=1 --type=incr --annotation=percona.com/backup-name=postgres-ash-vega-prod-repo1-incr-9qg9g

Also: stuck manual jobs remain Terminating indefinitely holding the internal.percona.com/keep-job finalizer after their CR is deleted.

Steps to reproduce

1. Deploy a PerconaPGCluster with operator 3.0.0 and backups.pgbackrest.repos[0].schedules set (e.g. incremental 20 0,3,9,12,15,18,21 * * *, S3 repo with dedicated repo host).
2. Watch a schedule slot: a CronJob job runs pgbackrest directly AND ~30s later an unannotated PerconaPGBackup CR is created; its reconcile spawns a <cluster>-backup-* job → lock collision on the repo host.
3. Let the manual job fail past its backoff limit (easy when the backup is long enough to overlap): its CR stays in Starting forever and all subsequent pg-backup CRs stop being reconciled (empty state).

Versions

• Operator: docker.io/percona/percona-postgresql-operator:3.0.0
• Database: PostgreSQL 18, percona/percona-pgbackrest:2.58.0-2 (pgBackRest 2.58.0)
• Kubernetes: v1.36.1+rke2r2 (RKE2, bare metal)
• Backup storage: S3-compatible object storage (path-style), dedicated repo host, archive-async=y

Anything else?

Production workaround we deployed: a cron "janitor" that (a) deletes scheduler-created CRs — no pgv2.percona.com/pgbackrest-backup-job-name annotation and name matching <cluster>-repo1-(incr|full)-[a-z0-9]{5} — before they spawn duplicate jobs, and (b) strips the orphaned keep-job finalizer from Terminating jobs. With the upstream CronJob as the single executor, every slot completes cleanly.

Suggested fixes:

• Gate the upstream CronJob creation when the in-process scheduler is active (or vice versa), or make the scheduler-created CR adopt the CronJob's running job in findBackupJob();
• Add a timeout/failure transition to the BackupStarting wait so a CR whose job disappeared cannot hold the lease forever.