Altinity / Altinity/clickhouse-backup
Streaming / continuous backups?
@ardigan6 is already working on this.
Since Oct 20, 2024.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 276
- Avg merge
- 18h 3m
- Merged PRs (30d)
- 26
Description
What we'd like to do is accept up to N minutes of loss (i.e. much less than the retention window of our queues) and run XXL single node CH instances with sharding / merging externally managed, since CH replicated mergetree is much slower than large nodes with plain mergetree, and inserts are >10x more ops replicated according to the docs.
However, this requires being able to bring up new nodes pretty quickly without a long rebuild process to handle failover / node replacement. We can then replay inserts very easily from the last row.
Right now backups seem too heavy to run every 60s / every N rows committed. We are self-hosters but I notice this is a feature gap in CH cloud too: backups are only daily.
Any plans to make this more efficient?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.