ClickHouse / ClickHouse/ClickHouse

Automatic MergeTree merge remains at progress=1 indefinitely and background merge scheduling stops

Open
#110,772 0 comments 0 reactions 0 assignees View on GitHub
comp-background-ops external potential bug
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

_No response_

### Describe what's wrong

On a single-node ClickHouse server, automatic MergeTree background merge
scheduling can become permanently stalled. A very small merge reaches
`progress = 1` with all rows read and written, but it remains in
`system.merges` for hours and does not disappear. While this state persists,
many MergeTree tables accumulate active parts and inserts are delayed or
rejected with `TOO_MANY_PARTS`.

This is not an `OPTIMIZE ... FINAL` operation. It is an automatic background
merge. There was no manual `SYSTEM STOP MERGES` or `SYSTEM START MERGES` around
the incident.

### Does it reproduce on the most recent release?

No

### How to reproduce

ClickHouse version: `26.6.1.1193-stable`.

Interface: native client / HTTP ingestion. The failure is in automatic server-side background merge scheduling, not in a client query.

Non-default settings:

```text
background_pool_size = 16
background_merges_mutations_concurrency_ratio = 2
```

Representative affected table, with identifiers anonymized:

```sql
CREATE TABLE default.scheduler_hosts
(
`cluster_name` String,
`sampling_time` DateTime,
`host_name` String,
`host_status` String,
`jl_u` Nullable(Decimal(10, 0)),
`max_slot` Nullable(Decimal(10, 0)),
`njobs` Nullable(Decimal(10, 0)),
`run_slot` Nullable(Decimal(10, 0)),
`sys_susp_slot` Nullable(Decimal(10, 0)),
`user_susp_slot` Nullable(Decimal(10, 0)),
`rsv_slot` Nullable(Decimal(10, 0)),
`interval_period` Nullable(Decimal(13, 0)),
`sampling_time_long` Nullable(Decimal(13, 0)),
`halt_num` Nullable(Decimal(13, 0)),
`gpus` Nullable(Decimal(13, 0)),
`gpu_slices` Nullable(Decimal(13, 0)),
`gpu_occ_rate` Nullable(Decimal(15, 4)) DEFAULT -1,
`gpu_num` Nullable(Decimal(15, 0)),
`gpu_used_num` Nullable(Decimal(15, 0))
)
ENGINE = MergeTree
PARTITION BY toDate(sampling_time)
ORDER BY (cluster_name, host_name, sampling_time)
SETTINGS index_granularity = 8192, parts_to_delay_insert = 200, parts_to_throw_insert = 500;
```

Observed triggering workload, not yet a deterministic minimal reproducer:

1. Use one ClickHouse server and the table above, together with several other MergeTree tables receiving small batches once per minute.
2. Continuously insert scheduler-host sampling data into the daily partition.
3. Monitor `system.merges` and active parts in `system.parts`.
4. After an undetermined runtime, an automatic merge can remain at `progress = 1` although `rows_read = rows_written`; active parts then accumulate across tables.

We can run a proposed minimal reproducer, enable a recommended diagnostic setting, or test a patch.

### Expected behavior

Once an automatic merge reports `progress = 1` and has written all output rows, it should finish and release the merge executor so background merge scheduling continues. Small, minute-level inserts should not accumulate parts indefinitely because one completed merge remains present.

### Error message and/or stacktrace

`system.merges` snapshot:

```text
database: default
table: scheduler_hosts
partition_id: 20260717
result_part_name: 20260717_16810_17066_64
num_parts: 5
total_size_bytes_compressed: approximately 13 KiB
rows_read: 257
rows_written: 257
progress: 1
elapsed: 19405 seconds
thread_id: 20520
```

The target merge thread (`LWP 20520`) from `pstack`:

```text
pthread_mutex_lock
std::mutex::lock
DB::MergeTreeBackgroundExecutor::threadFunction
```

Several background scheduler threads concurrently showed this path:

```text
DB::LockGuardWithStopWatch
DB::MergeTreeBackgroundExecutor::trySchedule
DB::BackgroundJobsAssignee::scheduleMergeMutateTask
DB::StorageMergeTree::scheduleDataProcessingJob
```

The full `pstack`, `system.merges`, `system.parts`, `system.events`, and server log excerpts are attached.

### Related issues and pull requests

_No response_

### Additional context

```text
OS: CentOS/RHEL 7 compatible, Linux kernel 3.10.0-123.el7.x86_64
Deployment: single ClickHouse server, no replication
Resources: 4 vCPU, 7.6 GiB RAM
```

Impact during this incident:

- A separate high-frequency state table reached about 501 active parts.
- Inserts began to be delayed and rejected: `DelayedInserts = 2038`, `RejectedInserts = 146`; errors included `TOO_MANY_PARTS`.
- CPU, memory, and disk were not saturated.
- SQL-level recovery attempts did not unblock the executor. A server restart recovered previous incidents, but this incident was intentionally left running while evidence was collected.

Questions:

1. Is this a known issue in 26.6.1.1193-stable or fixed in a later stable/LTS release?
2. Does the `threadFunction` and `trySchedule` mutex contention indicate a known deadlock/stall condition in `MergeTreeBackgroundExecutor`?
3. What additional diagnostic output is useful before restart, and is there a supported online recovery action for this executor state?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.