Altinity / Altinity/clickhouse-backup

upload/download transfers files within a part sequentially — add file-level parallelism (file_transfer_concurrency)

Open
#1,454 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.6k
Forks
276
Avg merge
18h 3m
Merged PRs (30d)
26

Description

Problem

UploadPath and DownloadPath in pkg/storage/general.go transfer the files of a part strictly sequentially:

  • DownloadPath (pkg/storage/general.go:699) walks the remote prefix and downloads one file at a time inside bd.Walk(...).
  • UploadPath (pkg/storage/general.go:766) iterates for _, filename := range files and calls PutFile one file at a time.

upload_concurrency / download_concurrency parallelize across parts/tables, but each part's files still move one-by-one. For wide MergeTree parts (dozens to hundreds of small/medium column files) against high-latency object storage (S3/GCS), per-file round-trip latency dominates and a single part transfers far below the available bandwidth. On multi-TiB backups with many wide tables this leaves most of the network idle.

Proposal

Add a file_transfer_concurrency option (general section, env FILE_TRANSFER_CONCURRENCY) and parallel variants UploadPathParallel / DownloadPathParallel (plus a manifest-aware DownloadPathParallelWithManifest) that transfer the files of a single part with N concurrent workers using errgroup:

  • Call sites in pkg/backup/upload.go and pkg/backup/download.go pick the parallel variant when file_transfer_concurrency > 1, otherwise the existing sequential code path runs unchanged.
  • Bandwidth throttling still honors upload_max_bytes_per_second / download_max_bytes_per_second as an aggregate cap: one shared bwlimit.Limiter is created per path-transfer and shared by all workers, so parallelism does not multiply the configured max speed.
  • Default file_transfer_concurrency: 1 preserves current behavior exactly (opt-in).

We run this in a production fork backing up multi-TiB ClickHouse clusters; file-level parallelism is the single biggest wall-clock win we measured for wide tables (parts with many files) on object storage.

I have a PR ready to submit.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read UploadPath and DownloadPath in pkg/storage/general.go first, then follow their call sites in pkg/backup/upload.go and pkg/backup/download.go. Check how the general configuration and existing bandwidth limits are wired before reviewing the proposed parallel variants. Done means the opt-in file concurrency works for uploads, downloads, and manifest-aware downloads while the default path and aggregate limits retain current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.