Aiven-Open / Aiven-Open/pghoard
Crash on .partial log files after recovery
- Linguagem predominante
- Python
- Estrelas
- 1.4k
- Forks
- 111
- Merge médio
- 1d 5h
- PRs com merge (30d)
- 2
Descrição
**How to reproduce**
Running a primary and replica with pghoard configured as `walreceiver`, terminate the primary, then promote the replica. In some cases where the state of logs was unclear when the primary goes down this will create a .partial file in the pg_xlog directory.
I tested the terminate-and-promote-to-master a few times on test databases and didn't see this happen, however we encountered it in production on a large, high traffic database which may have had something to do with the state of the xlog at the time.
**Expected Outcome**
The newly-promoted primary continues to back up xlog files.
**Actual Outcome**
Backups begin failing because pghoard doesn't know what to do with the .partial file.
Here is log output from when this happened:
```
127.0.0.1 - - [23/Mar/2017 22:40:18] "PUT /default/archive/00000001000000AC00000084.partial HTTP/1.1" 400 -
2017-03-23 22:40:19,616 WebServer Thread-23 ERROR HttpResponse 400: Unrecognized file '00000001000000AC00000084.partial' for archiving
/usr/local/bin/pghoard_postgres_command: ERROR: Archival failed with HTTP status 400
LOG: archive command failed with exit code 3
DETAIL: The failed archive command was: pghoard_postgres_command --mode archive --site default --xlog 00000001000000AC00000084.partial
```
This results in a pileup of xlog files and none being archived until the issue is resolved.
I was able to manually intervene and remove the `pg_xlog/archive_status/00000001000000AC00000084.partial.ready` file, causing postgres to stop trying to archive that file, but ideally pghoard would handle this better.
Thanks!
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Avaliação
Esta issue ainda não foi avaliada.