cloudnative-pg / cloudnative-pg/plugin-barman-cloud

ObjectStore’s maxParallel doesn’t actually increase the number of parallel uploads

未关闭
#820 4 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
bug
主要语言
Go
星标
192
派生
75
平均合并
1 天 16 小时
30 天内合并 PR
18

描述

Say you have 10 WAL files ready for archiving and maxParallel is set to 3.

* PG runs `archive_command` for WAL file 1:
* barman-plugin looks at ready files and run the archiver for WAL 1, 2 and 3.
* Those 3 files are uploaded to the archive.
* Command exits successfully and PG marks WAL 1 as done.
* Then PG runs `archive_command` for WAL file 2:
* barman-plugin looks are ready file and runs the archiver for WAL 2, 3 and 4
* WAL file 2 and 3 are already archived.
* WAL file 4 is uploaded to the archive.
* Command exits successfully and PG marks WAL 2 as done.
* And so on, PG archives WAL file 3 and barman-plugin uploads WAL file 5.

This results in always uploading one single WAL file at a time which is pretty slow and is contrary to what the doc says: “Number of WAL files to be […] archived in parallel”.

What I would expect to happen is:
* PG runs `archive_command` for WAL file 1:
* WAL file 1, 2 and 3 are uploaded
* WAL file 1 is marked as done
* PG runs `archive_command` for WAL file 2:
* WAL file 4, 5, 6 are uploaded
* WAL file 2 is marked as done
* And so on until there are no more files to upload and the `archive_command` simply exits successfully without doing anything.

I think this happens because `internalRun` does not pass WALs that have already been archived to `GatherReadyWALFiles`: https://github.com/cloudnative-pg/plugin-barman-cloud/blob/376e178ab5ea907aae50e6eaeb19215ec4326c53/internal/cnpgi/common/wal.go#L206.

贡献指南

打开贡献指南

调研方向

从 internal/cnpgi/common/wal.go 中的 internalRun 开始,检查它如何调用 GatherReadyWALFiles,并以 issue 中 ten-WAL、maxParallel=3 的场景作为参考。完成的标准是:连续的 archive_command 运行会以并行度大小的分组上传不同的批次,避免重复上传已归档的 WAL,并在没有剩余文件时成功退出。

由索引模型根据 Issue 内容生成。

评估

技术栈
go, postgresql
领域
backend, databases
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
62/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。