cloudnative-pg / cloudnative-pg/plugin-barman-cloud
ObjectStore’s maxParallel doesn’t actually increase the number of parallel uploads
- 主要言語
- Go
- スター
- 191
- フォーク
- 72
- 平均マージ
- 2日 21時間
- マージ済み PR(30日)
- 21
説明
Say you have 10 WAL files ready for archiving and maxParallel is set to 3.
* PG runs `archive_command` for WAL file 1:
* barman-plugin looks at ready files and run the archiver for WAL 1, 2 and 3.
* Those 3 files are uploaded to the archive.
* Command exits successfully and PG marks WAL 1 as done.
* Then PG runs `archive_command` for WAL file 2:
* barman-plugin looks are ready file and runs the archiver for WAL 2, 3 and 4
* WAL file 2 and 3 are already archived.
* WAL file 4 is uploaded to the archive.
* Command exits successfully and PG marks WAL 2 as done.
* And so on, PG archives WAL file 3 and barman-plugin uploads WAL file 5.
This results in always uploading one single WAL file at a time which is pretty slow and is contrary to what the doc says: “Number of WAL files to be […] archived in parallel”.
What I would expect to happen is:
* PG runs `archive_command` for WAL file 1:
* WAL file 1, 2 and 3 are uploaded
* WAL file 1 is marked as done
* PG runs `archive_command` for WAL file 2:
* WAL file 4, 5, 6 are uploaded
* WAL file 2 is marked as done
* And so on until there are no more files to upload and the `archive_command` simply exits successfully without doing anything.
I think this happens because `internalRun` does not pass WALs that have already been archived to `GatherReadyWALFiles`: https://github.com/cloudnative-pg/plugin-barman-cloud/blob/376e178ab5ea907aae50e6eaeb19215ec4326c53/internal/cnpgi/common/wal.go#L206.
コントリビューションガイド
調査の方向性
internal/cnpgi/common/wal.go の internalRun から始め、GatherReadyWALFiles をどのように呼び出しているかを調べてください。issue にある ten-WAL、maxParallel=3 のシナリオを指針にします。完了条件は、連続する archive_command の実行が異なるバッチを並列数単位のグループでアップロードし、アーカイブ済みの WAL を再アップロードせず、ファイルが残っていない場合に正常終了することです。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- go, postgresql
- 領域
- backend, databases
- issue の種類
- バグ
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 活発さ
- 静か
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 62/100