ClickHouse / ClickHouse/ClickHouse
Leftover nodes in ZK/Keeper from backup coordinator
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 17h 34m
- Merged PRs (30d)
- 479
Description
ClickHouse's backup coordinator creates a lot of nodes inside ZK/Keeper (depending on the number of files).
If something unexpected happens or CH server crashes while backup is in progress those nodes won't be cleaned up, leaving a lot of trash behind.
To verify if there are some leftover nodes you can check with
```
SELECT name
FROM system.zookeeper
WHERE path = '/clickhouse/backups' --- or whatever you set as the backup path
```
The resulting names should be `backup-some-uuid`.
By further checking the stored stage
```
SELECT name, time
FROM system.zookeeper
WHERE path = '/clickhouse/backups/backup-some-uuid/stage'
```
if we don't notice a name starting with `alive` it means we have a leftover node.
We should add a `SYSTEM` query to clean up those leftover nodes because it can dramatically increase the memory usage of ZK/Keeper if not cleaned up.
Contributor guide
Assessment
This issue has not been assessed yet.