logical-backup: The node was low on resource: ephemeral-storage.

The logical backup job fails because there isn't enough free space on the pod when the job runs.

• registry.opensource.zalan.do/acid/postgres-operator
• Azure K8s
• Production

I noticed what backup job fails. When I check pod logs I see errors:

kubectl describe po -n databases logical-backup-pgsql-manual-001-hbk28
Warning  Evicted              15m                kubelet            The node was low on resource: ephemeral-storage. Threshold quantity: 5217230516, available: 4087796Ki. Container logical-backup was using 18933736Ki, request is 0, has larger consumption of ephemeral-storage.

I run the backup job manually using the command

kubectl create job -n databases --from=cronjob/logical-backup-pgsql logical-backup-pgsql-manual-001

Then wait and I check the size of the backup. The full backup size is usually about 22GB.

kubectl exec -it -n databases logical-backup-pgsql-manual-001-48rmp -- du -hS /tmp/azure-backup.sql.gz
18G	/tmp/azure-backup.sql.gz

However, the total size of the filesystem is not enough for storing so much data.

kubectl exec -it -n databases logical-backup-pgsql-manual-001-48rmp -- df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          49G   41G  5.6G  89% /
tmpfs            64M     0   64M   0% /dev
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/sda3        49G   41G  5.6G  89% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           500M   12K  500M   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            32G     0   32G   0% /proc/acpi
tmpfs            32G     0   32G   0% /proc/scsi
tmpfs            32G     0   32G   0% /sys/firmware

Start by reviewing the logical-backup job configuration and reproduce the failure with the kubectl create job command shown in the issue. Compare the backup's temporary storage needs with the pod and node capacity; done means the full backup completes without the pod being evicted for ephemeral-storage.