Aiven-Open / Aiven-Open/pghoard
GCP 403 error with PGHoard 2.2.1
- Lenguaje dominante
- Python
- Estrellas
- 1.4k
- Forks
- 111
- Merge medio
- 1 d 5 h
- PR fusionados (30 d)
- 2
Descripción
I am trying to upload my backup data to GCP. When running PGHoard i am getting status code 401.
This error comes with version 2.2.1 of PGHoard and not version 2.1.1. When i am using PGHoard 2.2.1 and saving the data local the problem will not happen. I have not been testing other cloud providers (AWS etc) so i can't tell if this only happens in GCP but i am pretty sure its only there.
Setup:
Pre install:
```
sudo apt-get update
sudo apt-get install python3-pip
sudo apt install libpq-dev python3-dev
pip3 install psycopg2-binary
pip3 install google-api-python-client
pip3 install oauth2client
```
PGHoard install:
```
git clone https://github.com/aiven/pghoard
python3 setup.py bdist_egg
sudo python3 -m easy_install pghoard-2.2.1.dev65-py3.6.egg
```
PSQL:
```
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install postgresql-10
```
PSQL config:
```
wal_level = archive
max_wal_senders = 10
host replication pghoard 127.0.0.1/32 md5
CREATE USER pghoard PASSWORD 'PASSWORD_FOR_THE_PGHOARD_PSQL_USER' REPLICATION;
```
pghoard.json
```
{
"backup_location": "/var/lib/pghoard",
"syslog": true,
"compression": {
"algorithm": "snappy"
},
"backup_sites": {
"example-site": {
"active": true,
"version": 10,
"active_backup_mode": "pg_receivexlog",
"basebackup_count": 30,
"basebackup_interval_hours": 24,
"basebackup_mode": "pipe",
"basebackup_chunks_in_progress": 2,
"nodes": [
{
"host": "127.0.0.1",
"password": "PASSWORD_FOR_THE_PGHOARD_PSQL_USER",
"port": 5432,
"user": "pghoard"
}
],
"object_storage": {
"bucket_name": "NAME_OF_BUCKET",
"credential_file": "PATH/TO/THE/KEY.JSON",
"project_id": "PROJECT_ID",
"storage_type": "google"
},
"pg_bin_directory": "/usr/lib/postgresql/10/bin",
"pg_data_directory": "/var/lib/postgresql/10/main"
}
}
}
```
After making this setup i try to run PGHoard with the command:
```pghoard pghoard.json```
And gets the current error:
```
2021-09-10 12:21:53,638 pghoard MainThread ERROR Unexpected exception in PGHoard main loop
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 739, in run
self.handle_site(site, site_config)
File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 615, in handle_site
self.refresh_backup_list_and_delete_old(site)
File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 452, in refresh_backup_list_and_delete_old
basebackups = self.get_remote_basebackups_info(site)
File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 382, in get_remote_basebackups_info
storage = self.get_or_create_site_storage(site=site)
File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/pghoard.py", line 377, in get_or_create_site_storage
storage = get_transfer(storage_config)
File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/__init__.py", line 40, in get_transfer
return storage_class(**storage_config)
File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/object_storage/google.py", line 106, in __init__
self.bucket_name = self.get_or_create_bucket(bucket_name)
File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/object_storage/google.py", line 411, in get_or_create_bucket
self._retry_on_reset(request, request.execute)
File "/usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/object_storage/google.py", line 167, in _retry_on_reset
return action()
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/googleapiclient/http.py", line 937, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError:
2021-09-10 12:21:55,320 pghoard MainThread WARNING Quitting, signal: 2
```
When running PGHoard versionn 2.1.1 this error will not happen and will instead do as intended and upload the PSQL backup to my bucket on GCP
"Fix":
A way i have found around this is to either use version 2.1.1 of PGHoard instead of version 2.2.1 or another small "fix". If i change the line in the google.py (My path is /usr/local/lib/python3.6/dist-packages/pghoard-2.2.1.dev65-py3.6.egg/pghoard/rohmu/object_storage/google.py) and change line 132 which is
`return build("storage", "v1", http=http)`
to the line from PGHoard 2.1.1 which is on line 109
`return build("storage", "v1", credentials=self.google_creds)`
i get to upload the data to my GCP bucket. This will do that the HTTP object won't be used but the google credentials object. This means some features won't be able to be used so this fix won't be able to help everyone.
This is only a temporary fix and should not be the right way to do it but i have not been able to find the right way to do this.
I hope there is some smart people who are able to find the right way to fix this or just explain something i do wrong.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.