ClickHouse / ClickHouse/ClickHouse
Cleanup thread stops after ZooKeeper session expiry, parts accumulate
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
Personal project
### Describe what's wrong
The ReplicatedMergeTree cleanup thread silently stops processing outdated/inactive parts after a brief ZooKeeper session expiry. Parts accumulate indefinitely until the table hits `max_parts_in_total` limit.
**Symptoms:**
- Outdated parts count keeps growing indefinitely
- Table fails with `max_parts_in_total` or `parts_to_throw_insert` errors
- Cleanup works on some tables but stops on others (per-table issue)
- No errors in logs - cleanup thread stops silently
- Parts have `last_removal_attempt_time = '1970-01-01 00:00:00'`
**Root cause:** Race condition in `ReplicatedMergeTreeCleanupThread::run()` - returns early on `ZSESSIONEXPIRED` without rescheduling, and restarting thread may not restart it if ZK reconnects quickly.
### Does it reproduce on the most recent release?
Yes
### How to reproduce
1. Create a ReplicatedMergeTree table with high insert rate
2. Brief ZK session expiry occurs during cleanup (e.g., network blip)
3. ZK reconnects before restarting thread runs
4. Cleanup thread never runs again
**Identify with:**
```sql
SELECT
table,
count() as inactive_parts,
max(last_removal_attempt_time) as last_cleanup_attempt,
countIf(last_removal_attempt_time = toDateTime(0)) as never_seen_by_cleanup
FROM system.parts
WHERE NOT active
GROUP BY table
HAVING inactive_parts > 100;
```
If `never_seen_by_cleanup` grows while `last_cleanup_attempt` is stuck, you're hitting this bug.
### Expected behavior
Cleanup thread should continue processing parts after ZK reconnects.
### Error message and/or stacktrace
No errors - the thread stops silently.
### Additional context
**Workaround:** Restart ClickHouse server or `DETACH`/`ATTACH` the affected table.
**Proposed fix:** Modify `ZSESSIONEXPIRED` handler to reschedule with `max_cleanup_delay_period` instead of returning early. PR incoming.
- Affects: `ReplicatedMergeTree` tables
- Trigger: Brief ZooKeeper connectivity issues
- More likely with: High insert rates, network instability
Contributor guide
Assessment
This issue has not been assessed yet.