ClickHouse / ClickHouse/ClickHouse

Cleanup thread stops after ZooKeeper session expiry, parts accumulate

Open
#92,607 0 comments 0 reactions 0 assignees View on GitHub
comp-replication external
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

Personal project

### Describe what's wrong

The ReplicatedMergeTree cleanup thread silently stops processing outdated/inactive parts after a brief ZooKeeper session expiry. Parts accumulate indefinitely until the table hits `max_parts_in_total` limit.

**Symptoms:**
- Outdated parts count keeps growing indefinitely
- Table fails with `max_parts_in_total` or `parts_to_throw_insert` errors
- Cleanup works on some tables but stops on others (per-table issue)
- No errors in logs - cleanup thread stops silently
- Parts have `last_removal_attempt_time = '1970-01-01 00:00:00'`

**Root cause:** Race condition in `ReplicatedMergeTreeCleanupThread::run()` - returns early on `ZSESSIONEXPIRED` without rescheduling, and restarting thread may not restart it if ZK reconnects quickly.

### Does it reproduce on the most recent release?

Yes

### How to reproduce

1. Create a ReplicatedMergeTree table with high insert rate
2. Brief ZK session expiry occurs during cleanup (e.g., network blip)
3. ZK reconnects before restarting thread runs
4. Cleanup thread never runs again

**Identify with:**
```sql
SELECT
table,
count() as inactive_parts,
max(last_removal_attempt_time) as last_cleanup_attempt,
countIf(last_removal_attempt_time = toDateTime(0)) as never_seen_by_cleanup
FROM system.parts
WHERE NOT active
GROUP BY table
HAVING inactive_parts > 100;
```

If `never_seen_by_cleanup` grows while `last_cleanup_attempt` is stuck, you're hitting this bug.

### Expected behavior

Cleanup thread should continue processing parts after ZK reconnects.

### Error message and/or stacktrace

No errors - the thread stops silently.

### Additional context

**Workaround:** Restart ClickHouse server or `DETACH`/`ATTACH` the affected table.

**Proposed fix:** Modify `ZSESSIONEXPIRED` handler to reschedule with `max_cleanup_delay_period` instead of returning early. PR incoming.

- Affects: `ReplicatedMergeTree` tables
- Trigger: Brief ZooKeeper connectivity issues
- More likely with: High insert rates, network instability

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.