ClickHouse / ClickHouse/ClickHouse

AST fuzzer: server stuck above its memory limit after the fuzz run kills the job before artifacts upload (unattributable MEMORY_LIMIT_EXCEEDED failures)

Open
#111,892 3 comments 0 reactions 0 assignees View on GitHub
comp-ci-infrastructure
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

Requested by alexey-milovidov in https://github.com/ClickHouse/ClickHouse/pull/110074#issuecomment-5075859667.

### Failure mode

After an AST fuzzer run ends (fuzzer client killed at its time budget), the server can be stuck above its memory limit while idle. Exemplar: `AST fuzzer (amd_debug, targeted, old_compatibility)` on PR #110074, head `99589a86` ([GHA job 89472316051](https://github.com/ClickHouse/ClickHouse/actions/runs/30084913668/job/89472316051)):

- 60m fuzz (extended run) ends at 13:03:58 with fuzzer exit 137.
- Every post-fuzz `SELECT 1` probe is rejected: `Code: 241 ... (total) memory limit exceeded: would use 55.80 GiB (attempt to allocate chunk of 0.00 B), current RSS: 57.11 GiB, maximum: 46.32 GiB`.
- With zero queries running, tracked usage and RSS keep climbing (~110 MB/s): RSS 55.39 -> 57.71 GiB over the 21 s probe window. The cap is `max_server_memory_usage_to_ram_ratio = 0.75` of a 61.76 GiB runner.
- ~23 s into the probe loop the job is canceled (`The operation was canceled.`), before `status.tsv` is written and before any artifact upload. No server.log, no fuzzer.log, no CIDB row: the offending query history is lost, so the memory growth cannot be attributed.

The harness contributes on two points (`ci/jobs/scripts/fuzzer/run-fuzzer.sh`):

1. The post-fuzz liveness loop treats `MEMORY_LIMIT_EXCEEDED` as "alive, give it time to reclaim" and retries up to ~2 min. It cannot distinguish a transient post-query spike from tracked memory pinned above the cap while idle.
2. If the loop exits with `server_died=0`, the script attempts a graceful `clickhouse stop` with an unbounded `wait`. Graceful shutdown of a memory-saturated debug server is unreliable, and RSS keeps growing toward physical RAM, so the job hangs until something kills it (workflow cancel, runner death, or kernel OOM), destroying the evidence.

### Frequency (CIDB, 2026-07-25)

Vanished-job runs (one targeted twin has CIDB rows for a SHA, the other has none because the job died before reporting), last 7 days: 4 missing `old_compatibility` + 6 missing plain `targeted` - about 1/day per variant. The mode is symmetric across variants (the harness path is shared by all AST fuzzer and BuzzHouse variants), not specific to `old_compatibility`.

```sql
SELECT pull_request_number, left(commit_sha,12) sha, min(check_start_time) t
FROM default.checks
WHERE check_name = 'AST fuzzer (amd_debug, targeted)'
AND check_start_time > now() - INTERVAL 7 DAY
AND commit_sha NOT IN (
SELECT DISTINCT commit_sha FROM default.checks
WHERE check_name = 'AST fuzzer (amd_debug, targeted, old_compatibility)'
AND check_start_time > now() - INTERVAL 8 DAY)
GROUP BY 1,2 ORDER BY t DESC
```
(and the symmetric query with the two check names swapped)

Note the decomposition: the bulk of `old_compatibility` FAIL rows across unrelated PRs (28 rows / 24 distinct failure names in the trailing 3 days as of filing; ~90 rows in the 3-day window around 2026-07-24) are attributed per-query crashes with distinct STIDs (count 1-5 each) - pre-existing trunk bugs surfaced by fuzzing, individually actionable per STID, not one shared bug. This issue tracks only the unattributable idle-memory mode above.

### Harness fix

Fix PR: link to follow shortly.

It bounds every post-budget stage of `run-fuzzer.sh` (client reap, liveness probing, teardown) and makes the stuck state a first-class failure: the probe loop detects persistent global-tracker rejection (two tiers: 12 consecutive rejections with host `MemAvailable` under 4 GiB, or 60 consecutive regardless), records a `server_memory_stuck.txt` marker with the tracker line, SIGKILLs the server, and lets the job complete its normal artifact upload with a deterministic FAIL row (`Server unresponsive: memory limit exceeded`). Reap/teardown escalations write a `harness_watchdog.txt` and surface as ERROR. Fuzz budgets and the client kill grace are unchanged; fail fast with evidence instead of hanging until the box dies.

### Open follow-up (kept open after the harness fix merges)

Root-cause the idle post-fuzz memory growth from the artifacts the fix makes available (server.log MemoryTracker lines + fuzzer.log query history of stuck runs). Candidates: accumulated fuzzed Memory/Set/Join table state, background merges/mutations of fuzzed tables (merge memory is not bounded by query limits, see #107410), system-log flush feedback under memory pressure.

Related: #109590 (a distinct sub-mode: fuzzer client exits 241 while the server stays healthy - job-classification fix only).

Contributor guide

Open the contributing guide

Research direction

Start with ci/jobs/scripts/fuzzer/run-fuzzer.sh and the artifacts produced by the harness fix, especially server.log and fuzzer.log from stuck runs. Compare MemoryTracker lines with the recorded query history, then investigate the stated candidates: accumulated fuzzed table state, background merges or mutations, and system-log flushing. Done means identifying and fixing the idle post-fuzz memory growth, with a reproducible test or evidence from affected runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, shell, sql
Domain
ci-cd, databases, performance, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.