cloudnative-pg / cloudnative-pg/plugin-barman-cloud

ObjectStore’s maxParallel doesn’t actually increase the number of parallel uploads

Open
#820 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
191
Forks
72
Avg merge
1d 16h
Merged PRs (30d)
18

Description

Say you have 10 WAL files ready for archiving and maxParallel is set to 3.

* PG runs `archive_command` for WAL file 1:
* barman-plugin looks at ready files and run the archiver for WAL 1, 2 and 3.
* Those 3 files are uploaded to the archive.
* Command exits successfully and PG marks WAL 1 as done.
* Then PG runs `archive_command` for WAL file 2:
* barman-plugin looks are ready file and runs the archiver for WAL 2, 3 and 4
* WAL file 2 and 3 are already archived.
* WAL file 4 is uploaded to the archive.
* Command exits successfully and PG marks WAL 2 as done.
* And so on, PG archives WAL file 3 and barman-plugin uploads WAL file 5.

This results in always uploading one single WAL file at a time which is pretty slow and is contrary to what the doc says: “Number of WAL files to be […] archived in parallel”.

What I would expect to happen is:
* PG runs `archive_command` for WAL file 1:
* WAL file 1, 2 and 3 are uploaded
* WAL file 1 is marked as done
* PG runs `archive_command` for WAL file 2:
* WAL file 4, 5, 6 are uploaded
* WAL file 2 is marked as done
* And so on until there are no more files to upload and the `archive_command` simply exits successfully without doing anything.

I think this happens because `internalRun` does not pass WALs that have already been archived to `GatherReadyWALFiles`: https://github.com/cloudnative-pg/plugin-barman-cloud/blob/376e178ab5ea907aae50e6eaeb19215ec4326c53/internal/cnpgi/common/wal.go#L206.

Contributor guide

Open the contributing guide

Research direction

Start at internalRun in internal/cnpgi/common/wal.go and inspect how it calls GatherReadyWALFiles, using the ten-WAL, maxParallel=3 scenario from the issue as a guide. Done means successive archive_command runs upload distinct batches in parallel-sized groups, avoid re-uploading archived WALs, and exit successfully when no files remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.