cloudnative-pg / cloudnative-pg/plugin-barman-cloud

ObjectStore’s maxParallel doesn’t actually increase the number of parallel uploads

Ouverte
#820 4 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
bug
Langage dominant
Go
Étoiles
191
Forks
72
Merge moyen
2 j 21 h
PR mergées (30 j)
21

Description

Say you have 10 WAL files ready for archiving and maxParallel is set to 3.

* PG runs `archive_command` for WAL file 1:
* barman-plugin looks at ready files and run the archiver for WAL 1, 2 and 3.
* Those 3 files are uploaded to the archive.
* Command exits successfully and PG marks WAL 1 as done.
* Then PG runs `archive_command` for WAL file 2:
* barman-plugin looks are ready file and runs the archiver for WAL 2, 3 and 4
* WAL file 2 and 3 are already archived.
* WAL file 4 is uploaded to the archive.
* Command exits successfully and PG marks WAL 2 as done.
* And so on, PG archives WAL file 3 and barman-plugin uploads WAL file 5.

This results in always uploading one single WAL file at a time which is pretty slow and is contrary to what the doc says: “Number of WAL files to be […] archived in parallel”.

What I would expect to happen is:
* PG runs `archive_command` for WAL file 1:
* WAL file 1, 2 and 3 are uploaded
* WAL file 1 is marked as done
* PG runs `archive_command` for WAL file 2:
* WAL file 4, 5, 6 are uploaded
* WAL file 2 is marked as done
* And so on until there are no more files to upload and the `archive_command` simply exits successfully without doing anything.

I think this happens because `internalRun` does not pass WALs that have already been archived to `GatherReadyWALFiles`: https://github.com/cloudnative-pg/plugin-barman-cloud/blob/376e178ab5ea907aae50e6eaeb19215ec4326c53/internal/cnpgi/common/wal.go#L206.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par internalRun dans internal/cnpgi/common/wal.go et examinez comment il appelle GatherReadyWALFiles, en vous servant comme guide du scénario de l’issue avec dix WAL et maxParallel=3. Le travail est terminé lorsque des exécutions successives de archive_command téléversent des lots distincts par groupes de la taille du parallélisme, évitent de téléverser à nouveau les WAL archivés et se terminent avec succès lorsqu’il ne reste plus de fichiers.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
go, postgresql
Domaine
backend, databases
Type d'issue
Bug
Difficulté
3/5
Temps estimé
1-2 jours
Activité
Calme
Clarté
Plutôt claire
Accessibilité débutants
62/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.