Aiven-Open / Aiven-Open/pghoard

Crash on .partial log files after recovery

Đang mở
#195 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
1.4k
Fork
111
Merge trung bình
1 ngày 5 giờ
Pull request đã merge (30 ngày)
2

Mô tả

**How to reproduce**
Running a primary and replica with pghoard configured as `walreceiver`, terminate the primary, then promote the replica. In some cases where the state of logs was unclear when the primary goes down this will create a .partial file in the pg_xlog directory.

I tested the terminate-and-promote-to-master a few times on test databases and didn't see this happen, however we encountered it in production on a large, high traffic database which may have had something to do with the state of the xlog at the time.

**Expected Outcome**
The newly-promoted primary continues to back up xlog files.

**Actual Outcome**
Backups begin failing because pghoard doesn't know what to do with the .partial file.

Here is log output from when this happened:
```
127.0.0.1 - - [23/Mar/2017 22:40:18] "PUT /default/archive/00000001000000AC00000084.partial HTTP/1.1" 400 -
2017-03-23 22:40:19,616 WebServer Thread-23 ERROR HttpResponse 400: Unrecognized file '00000001000000AC00000084.partial' for archiving
/usr/local/bin/pghoard_postgres_command: ERROR: Archival failed with HTTP status 400
LOG: archive command failed with exit code 3
DETAIL: The failed archive command was: pghoard_postgres_command --mode archive --site default --xlog 00000001000000AC00000084.partial
```

This results in a pileup of xlog files and none being archived until the issue is resolved.

I was able to manually intervene and remove the `pg_xlog/archive_status/00000001000000AC00000084.partial.ready` file, causing postgres to stop trying to archive that file, but ideally pghoard would handle this better.

Thanks!

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.