Aiven-Open / Aiven-Open/pghoard
Handle gap in remote xlog sequence
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 111
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 2
Description
Sometimes pghoard failed to upload some wal segments. Then pghoard restart and upload new wal segments. In this case, on remote sotrage, there is a gap in wal segment sequence.
I have implemented some new metrics to detects this case:
* total wal segments: total wal segments on remote storage
* continious wal segments: total wal segments without gap starting from last uploaded
* valid base backup: total basebackup starting from last uploaded without wal segments gap
* missing wal segments: Missing wal segment on remote storage between basebackups
* missing wal segments at end: Missing wal segment near the master position, should not go higher than 1. Replication lag for remote storage
In the following example:
* total wal segments: 74
* continious wal segments: 47
* valid base backup: 2
* missing wal segments: 6
* missing wal segments at end: 2
```
Legend:
#: basebackup
-: wal segment on remote storage
x: missing wal segment on remote storage
|: current master position
0: wal segment in progress in pghoard (receive from pg, compress, encrypt, upload)
valid basebackup valid basebackup
↓ ↓
#----------#--------------#---xxxxxx------------#-----------------#------------------00|
↑ ↑↑↑↑↑↑↑ ↑↑↑
other valid wal segments |||||||----------- continious wal segment ------------|||
|||||| ||
missing wal segments missing wal segment at end
```
To implements those metrics pghoard need to list files uploaded on remote storage and keep this list up to date.
Scanning remote storage (only on startup) can also improve cleanup because when there is a gap in wal sequence pghoard stop to delete useless wal segments. With the list of uploaded files we can delete all wal segments before the first basebackup even with gap.
https://github.com/aiven/pghoard/blob/master/pghoard/pghoard.py#L255-L258
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.