hapostgres / hapostgres/pg_auto_failover

Configuration parameter in pg_autoctl.cfg for the ability to clear PGDATA before running pg_basebackup

Open
#1,024 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
1.4k
Forks
142
Avg merge
5h 8m
Merged PRs (30d)
1

Description

Good afternoon.

As far as I can see from the behavior of the cluster and the source code, if after switchover/failover the replica cannot be connected using pg_rewind and the replica is recreated using pg_basebackup. At the same time, PGDATA on the replica is not cleared until the procedure for copying data from the wizard to the backup_directory directory specified in pg_autoctl.cfg is completed. After the pg_basebackup copy is successfully completed, the PGDATA is cleared and the backup_directory data is moved to the PGDATA location:
```
/*
* Call pg_basebackup, using a temporary directory for the duration of the data
* transfer.
*/
bool
pg_basebackup(const char *pgdata,
const char *pg_ctl,
ReplicationSource *replicationSource)
{

...

returnCode = program.returnCode;
free_program(&program);

if (returnCode != 0)
{
log_error("Failed to run pg_basebackup: exit code %d", returnCode);
return false;
}

/* replace $pgdata with the backup directory */
if (directory_exists(pgdata))
{
if (!rmtree(pgdata, true))
{
log_error("Failed to remove directory \"%s\": %m", pgdata);
return false;
}
}

log_debug("mv \"%s\" \"%s\"", replicationSource->backupDir, pgdata);

if (rename(replicationSource->backupDir, pgdata) != 0)
{
log_error(
"Failed to install pg_basebackup dir " " \"%s\" in \"%s\": %m",
replicationSource->backupDir, pgdata);
return false;
}

return true;
}
```
This behavior can lead to the fact that disk space in the pg_basebackup process will run out if there is a lot of data and there is not enough disk space for PGDATA and a fresh copy from primary.
This is a particularly unpleasant situation with a failover, when we hope that automation will bring the cluster back to normal after changing server roles.

It would be convenient to have a parameter in the pg_autoctl.cfg configuration file, setting it to true, PGDATA on this server was cleared before running pg_basebackup (if pg_rewind did not help) during switchover/failover. This would solve the problem and allow for a choice of behavior.

Thank you.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.