Altinity / Altinity/clickhouse-backup
upload/download transfers files within a part sequentially — add file-level parallelism (file_transfer_concurrency)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 276
- Avg merge
- 18h 3m
- Merged PRs (30d)
- 26
Description
Problem
UploadPath and DownloadPath in pkg/storage/general.go transfer the files of a part strictly sequentially:
DownloadPath(pkg/storage/general.go:699) walks the remote prefix and downloads one file at a time insidebd.Walk(...).UploadPath(pkg/storage/general.go:766) iteratesfor _, filename := range filesand callsPutFileone file at a time.
upload_concurrency / download_concurrency parallelize across parts/tables, but each part's files still move one-by-one. For wide MergeTree parts (dozens to hundreds of small/medium column files) against high-latency object storage (S3/GCS), per-file round-trip latency dominates and a single part transfers far below the available bandwidth. On multi-TiB backups with many wide tables this leaves most of the network idle.
Proposal
Add a file_transfer_concurrency option (general section, env FILE_TRANSFER_CONCURRENCY) and parallel variants UploadPathParallel / DownloadPathParallel (plus a manifest-aware DownloadPathParallelWithManifest) that transfer the files of a single part with N concurrent workers using errgroup:
- Call sites in
pkg/backup/upload.goandpkg/backup/download.gopick the parallel variant whenfile_transfer_concurrency > 1, otherwise the existing sequential code path runs unchanged. - Bandwidth throttling still honors
upload_max_bytes_per_second/download_max_bytes_per_secondas an aggregate cap: one sharedbwlimit.Limiteris created per path-transfer and shared by all workers, so parallelism does not multiply the configured max speed. - Default
file_transfer_concurrency: 1preserves current behavior exactly (opt-in).
We run this in a production fork backing up multi-TiB ClickHouse clusters; file-level parallelism is the single biggest wall-clock win we measured for wide tables (parts with many files) on object storage.
I have a PR ready to submit.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read UploadPath and DownloadPath in pkg/storage/general.go first, then follow their call sites in pkg/backup/upload.go and pkg/backup/download.go. Check how the general configuration and existing bandwidth limits are wired before reviewing the proposed parallel variants. Done means the opt-in file concurrency works for uploads, downloads, and manifest-aware downloads while the default path and aggregate limits retain current behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100