cloudnative-pg / cloudnative-pg/plugin-barman-cloud

ObjectStore’s maxParallel doesn’t actually increase the number of parallel uploads

Aperta
#820 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug
Lingua principale
Go
Stelle
191
Fork
72
Merge medio
2g 21h
PR unite (30g)
21

Descrizione

Say you have 10 WAL files ready for archiving and maxParallel is set to 3.

* PG runs `archive_command` for WAL file 1:
* barman-plugin looks at ready files and run the archiver for WAL 1, 2 and 3.
* Those 3 files are uploaded to the archive.
* Command exits successfully and PG marks WAL 1 as done.
* Then PG runs `archive_command` for WAL file 2:
* barman-plugin looks are ready file and runs the archiver for WAL 2, 3 and 4
* WAL file 2 and 3 are already archived.
* WAL file 4 is uploaded to the archive.
* Command exits successfully and PG marks WAL 2 as done.
* And so on, PG archives WAL file 3 and barman-plugin uploads WAL file 5.

This results in always uploading one single WAL file at a time which is pretty slow and is contrary to what the doc says: “Number of WAL files to be […] archived in parallel”.

What I would expect to happen is:
* PG runs `archive_command` for WAL file 1:
* WAL file 1, 2 and 3 are uploaded
* WAL file 1 is marked as done
* PG runs `archive_command` for WAL file 2:
* WAL file 4, 5, 6 are uploaded
* WAL file 2 is marked as done
* And so on until there are no more files to upload and the `archive_command` simply exits successfully without doing anything.

I think this happens because `internalRun` does not pass WALs that have already been archived to `GatherReadyWALFiles`: https://github.com/cloudnative-pg/plugin-barman-cloud/blob/376e178ab5ea907aae50e6eaeb19215ec4326c53/internal/cnpgi/common/wal.go#L206.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.