ClickHouse / ClickHouse/ClickHouse
Automatic MergeTree merge remains at progress=1 indefinitely and background merge scheduling stops
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Describe what's wrong
On a single-node ClickHouse server, automatic MergeTree background merge
scheduling can become permanently stalled. A very small merge reaches
`progress = 1` with all rows read and written, but it remains in
`system.merges` for hours and does not disappear. While this state persists,
many MergeTree tables accumulate active parts and inserts are delayed or
rejected with `TOO_MANY_PARTS`.
This is not an `OPTIMIZE ... FINAL` operation. It is an automatic background
merge. There was no manual `SYSTEM STOP MERGES` or `SYSTEM START MERGES` around
the incident.
### Does it reproduce on the most recent release?
No
### How to reproduce
ClickHouse version: `26.6.1.1193-stable`.
Interface: native client / HTTP ingestion. The failure is in automatic server-side background merge scheduling, not in a client query.
Non-default settings:
```text
background_pool_size = 16
background_merges_mutations_concurrency_ratio = 2
```
Representative affected table, with identifiers anonymized:
```sql
CREATE TABLE default.scheduler_hosts
(
`cluster_name` String,
`sampling_time` DateTime,
`host_name` String,
`host_status` String,
`jl_u` Nullable(Decimal(10, 0)),
`max_slot` Nullable(Decimal(10, 0)),
`njobs` Nullable(Decimal(10, 0)),
`run_slot` Nullable(Decimal(10, 0)),
`sys_susp_slot` Nullable(Decimal(10, 0)),
`user_susp_slot` Nullable(Decimal(10, 0)),
`rsv_slot` Nullable(Decimal(10, 0)),
`interval_period` Nullable(Decimal(13, 0)),
`sampling_time_long` Nullable(Decimal(13, 0)),
`halt_num` Nullable(Decimal(13, 0)),
`gpus` Nullable(Decimal(13, 0)),
`gpu_slices` Nullable(Decimal(13, 0)),
`gpu_occ_rate` Nullable(Decimal(15, 4)) DEFAULT -1,
`gpu_num` Nullable(Decimal(15, 0)),
`gpu_used_num` Nullable(Decimal(15, 0))
)
ENGINE = MergeTree
PARTITION BY toDate(sampling_time)
ORDER BY (cluster_name, host_name, sampling_time)
SETTINGS index_granularity = 8192, parts_to_delay_insert = 200, parts_to_throw_insert = 500;
```
Observed triggering workload, not yet a deterministic minimal reproducer:
1. Use one ClickHouse server and the table above, together with several other MergeTree tables receiving small batches once per minute.
2. Continuously insert scheduler-host sampling data into the daily partition.
3. Monitor `system.merges` and active parts in `system.parts`.
4. After an undetermined runtime, an automatic merge can remain at `progress = 1` although `rows_read = rows_written`; active parts then accumulate across tables.
We can run a proposed minimal reproducer, enable a recommended diagnostic setting, or test a patch.
### Expected behavior
Once an automatic merge reports `progress = 1` and has written all output rows, it should finish and release the merge executor so background merge scheduling continues. Small, minute-level inserts should not accumulate parts indefinitely because one completed merge remains present.
### Error message and/or stacktrace
`system.merges` snapshot:
```text
database: default
table: scheduler_hosts
partition_id: 20260717
result_part_name: 20260717_16810_17066_64
num_parts: 5
total_size_bytes_compressed: approximately 13 KiB
rows_read: 257
rows_written: 257
progress: 1
elapsed: 19405 seconds
thread_id: 20520
```
The target merge thread (`LWP 20520`) from `pstack`:
```text
pthread_mutex_lock
std::mutex::lock
DB::MergeTreeBackgroundExecutor::threadFunction
```
Several background scheduler threads concurrently showed this path:
```text
DB::LockGuardWithStopWatch
DB::MergeTreeBackgroundExecutor::trySchedule
DB::BackgroundJobsAssignee::scheduleMergeMutateTask
DB::StorageMergeTree::scheduleDataProcessingJob
```
The full `pstack`, `system.merges`, `system.parts`, `system.events`, and server log excerpts are attached.
### Related issues and pull requests
_No response_
### Additional context
```text
OS: CentOS/RHEL 7 compatible, Linux kernel 3.10.0-123.el7.x86_64
Deployment: single ClickHouse server, no replication
Resources: 4 vCPU, 7.6 GiB RAM
```
Impact during this incident:
- A separate high-frequency state table reached about 501 active parts.
- Inserts began to be delayed and rejected: `DelayedInserts = 2038`, `RejectedInserts = 146`; errors included `TOO_MANY_PARTS`.
- CPU, memory, and disk were not saturated.
- SQL-level recovery attempts did not unblock the executor. A server restart recovered previous incidents, but this incident was intentionally left running while evidence was collected.
Questions:
1. Is this a known issue in 26.6.1.1193-stable or fixed in a later stable/LTS release?
2. Does the `threadFunction` and `trySchedule` mutex contention indicate a known deadlock/stall condition in `MergeTreeBackgroundExecutor`?
3. What additional diagnostic output is useful before restart, and is there a supported online recovery action for this executor state?
Contributor guide
Assessment
This issue has not been assessed yet.