Aiven-Open / Aiven-Open/pghoard

Restoring from GCS takes too long

Aberta
#358 3 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
1.4k
Forks
111
Merge médio
1d 5h
PRs com merge (30d)
2

Descrição

Hello,

I'm trying to restore a postgres basebackup from Google Cloud Storage to Google Compute instance in the same zone.
I cannot figure out why, but `pghoard_restore` command needs ~106m to download and extract basebackup 18GB of size (48GB original size) while `gsutil` takes ~3 minutes only to download the same file.

During `pghoard_restore` avg network throughput is ~40MBit/s while `gsutil` turns it to `1.1GBit/s`.
What am I doing wrong? How can I tune the restore process somehow?

Pghoard config:
```json
{
"backup_location": "./metadata",
"backup_sites": {
"postgres-01": {
"active_backup_mode": "pg_receivexlog",
"basebackup_hour": 5,
"basebackup_interval_hours": 24,
"basebackup_minute": 40,
"pg_data_directory": "/var/lib/pgsql/9.5/data",
"nodes": [
{
"host": "postgres-01",
"user": "pghoard",
"password": "secret",
"port": 5432
}
],
"object_storage": {
"storage_type": "google",
"project_id": "postgres-dev",
"bucket_name": "test-pghoard"
}
}
}
}
```

Restore command:
```
pghoard_restore get-basebackup --config pghoard.json --recovery-target-time 2019-07-04T14:45:03+00:00 --restore-to-master --overwrite --target-dir /var/lib/pgsql/9.5/data/
```

Output:
```
Found 1 applicable basebackup

Basebackup Backup size Orig size Start time
---------------------------------------- ----------- ----------- --------------------
postgres-01/basebackup/2019-07-09_05-39_0 17687 MB 48779 MB 2019-07-09T05:41:38Z
metadata: {'backup-decision-time': '2019-07-09T05:39:45.442175+00:00', 'backup-reason': 'scheduled', 'normalized-backup-time': '2019-07-08T05:40:00+00:00', 'start-wal-segment': '000000040000C1120000009E', 'pg-version': '90504', 'compression-algorithm': 'snappy', 'compression-level': '0', 'original-file-size': '51148804096', 'host': 'postgres-01'}

Selecting 'postgres-01/basebackup/2019-07-09_05-39_0' for restore
2019-07-09 16:08:09,4699ChunkFetcher7687Thread-1 INFO Processing of 'postgres-01/basebackup/2019-07-09_05-39_0' completed successfully
Download progress: 100.00% (17687 / 17687 MiB)
Basebackup restoration complete.
You can start PostgreSQL by running pg_ctl -D /var/lib/pgsql/9.5/data/ start
On systemd based systems you can run systemctl start postgresql
On SYSV Init based systems you can run /etc/init.d/postgresql start
```

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.