ClickHouse / ClickHouse/ClickHouse

Leftover nodes in ZK/Keeper from backup coordinator

Open
#46,211 2 comments 2 reactions 1 assignee Assigned to @nikitamikhaylov View on GitHub
comp-backup unfinished code
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
17h 34m
Merged PRs (30d)
479

Description

ClickHouse's backup coordinator creates a lot of nodes inside ZK/Keeper (depending on the number of files).
If something unexpected happens or CH server crashes while backup is in progress those nodes won't be cleaned up, leaving a lot of trash behind.

To verify if there are some leftover nodes you can check with

```
SELECT name
FROM system.zookeeper
WHERE path = '/clickhouse/backups' --- or whatever you set as the backup path
```

The resulting names should be `backup-some-uuid`.

By further checking the stored stage
```
SELECT name, time
FROM system.zookeeper
WHERE path = '/clickhouse/backups/backup-some-uuid/stage'
```
if we don't notice a name starting with `alive` it means we have a leftover node.

We should add a `SYSTEM` query to clean up those leftover nodes because it can dramatically increase the memory usage of ZK/Keeper if not cleaned up.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.