Altinity / Altinity/clickhouse-backup

Streaming / continuous backups?

Open
#1,030 3 comments 0 reactions 1 assignee View on GitHub

@ardigan6 is already working on this.

Since Oct 20, 2024.

Dominant language
Go
Stars
1.6k
Forks
276
Avg merge
18h 3m
Merged PRs (30d)
26

Description

What we'd like to do is accept up to N minutes of loss (i.e. much less than the retention window of our queues) and run XXL single node CH instances with sharding / merging externally managed, since CH replicated mergetree is much slower than large nodes with plain mergetree, and inserts are >10x more ops replicated according to the docs.

However, this requires being able to bring up new nodes pretty quickly without a long rebuild process to handle failover / node replacement. We can then replay inserts very easily from the last row.

Right now backups seem too heavy to run every 60s / every N rows committed. We are self-hosters but I notice this is a feature gap in CH cloud too: backups are only daily.

Any plans to make this more efficient?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.