Aiven-Open / Aiven-Open/pghoard

Crash on .partial log files after recovery

未關閉
#195 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
1.4k
分支
111
平均合併
1 天 5 小時
30 天內合併 PR
2

描述

**How to reproduce**
Running a primary and replica with pghoard configured as `walreceiver`, terminate the primary, then promote the replica. In some cases where the state of logs was unclear when the primary goes down this will create a .partial file in the pg_xlog directory.

I tested the terminate-and-promote-to-master a few times on test databases and didn't see this happen, however we encountered it in production on a large, high traffic database which may have had something to do with the state of the xlog at the time.

**Expected Outcome**
The newly-promoted primary continues to back up xlog files.

**Actual Outcome**
Backups begin failing because pghoard doesn't know what to do with the .partial file.

Here is log output from when this happened:
```
127.0.0.1 - - [23/Mar/2017 22:40:18] "PUT /default/archive/00000001000000AC00000084.partial HTTP/1.1" 400 -
2017-03-23 22:40:19,616 WebServer Thread-23 ERROR HttpResponse 400: Unrecognized file '00000001000000AC00000084.partial' for archiving
/usr/local/bin/pghoard_postgres_command: ERROR: Archival failed with HTTP status 400
LOG: archive command failed with exit code 3
DETAIL: The failed archive command was: pghoard_postgres_command --mode archive --site default --xlog 00000001000000AC00000084.partial
```

This results in a pileup of xlog files and none being archived until the issue is resolved.

I was able to manually intervene and remove the `pg_xlog/archive_status/00000001000000AC00000084.partial.ready` file, causing postgres to stop trying to archive that file, but ideally pghoard would handle this better.

Thanks!

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。