hapostgres / hapostgres/pg_auto_failover
Configuration parameter in pg_autoctl.cfg for the ability to clear PGDATA before running pg_basebackup
- Dominant language
- C
- Stars
- 1.4k
- Forks
- 142
- Avg merge
- 5h 8m
- Merged PRs (30d)
- 1
Description
Good afternoon.
As far as I can see from the behavior of the cluster and the source code, if after switchover/failover the replica cannot be connected using pg_rewind and the replica is recreated using pg_basebackup. At the same time, PGDATA on the replica is not cleared until the procedure for copying data from the wizard to the backup_directory directory specified in pg_autoctl.cfg is completed. After the pg_basebackup copy is successfully completed, the PGDATA is cleared and the backup_directory data is moved to the PGDATA location:
```
/*
* Call pg_basebackup, using a temporary directory for the duration of the data
* transfer.
*/
bool
pg_basebackup(const char *pgdata,
const char *pg_ctl,
ReplicationSource *replicationSource)
{
...
returnCode = program.returnCode;
free_program(&program);
if (returnCode != 0)
{
log_error("Failed to run pg_basebackup: exit code %d", returnCode);
return false;
}
/* replace $pgdata with the backup directory */
if (directory_exists(pgdata))
{
if (!rmtree(pgdata, true))
{
log_error("Failed to remove directory \"%s\": %m", pgdata);
return false;
}
}
log_debug("mv \"%s\" \"%s\"", replicationSource->backupDir, pgdata);
if (rename(replicationSource->backupDir, pgdata) != 0)
{
log_error(
"Failed to install pg_basebackup dir " " \"%s\" in \"%s\": %m",
replicationSource->backupDir, pgdata);
return false;
}
return true;
}
```
This behavior can lead to the fact that disk space in the pg_basebackup process will run out if there is a lot of data and there is not enough disk space for PGDATA and a fresh copy from primary.
This is a particularly unpleasant situation with a failover, when we hope that automation will bring the cluster back to normal after changing server roles.
It would be convenient to have a parameter in the pg_autoctl.cfg configuration file, setting it to true, PGDATA on this server was cleared before running pg_basebackup (if pg_rewind did not help) during switchover/failover. This would solve the problem and allow for a choice of behavior.
Thank you.
Contributor guide
Assessment
This issue has not been assessed yet.