ClickHouse / ClickHouse/ClickHouse
AST fuzzer: server stuck above its memory limit after the fuzz run kills the job before artifacts upload (unattributable MEMORY_LIMIT_EXCEEDED failures)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
Requested by alexey-milovidov in https://github.com/ClickHouse/ClickHouse/pull/110074#issuecomment-5075859667.
### Failure mode
After an AST fuzzer run ends (fuzzer client killed at its time budget), the server can be stuck above its memory limit while idle. Exemplar: `AST fuzzer (amd_debug, targeted, old_compatibility)` on PR #110074, head `99589a86` ([GHA job 89472316051](https://github.com/ClickHouse/ClickHouse/actions/runs/30084913668/job/89472316051)):
- 60m fuzz (extended run) ends at 13:03:58 with fuzzer exit 137.
- Every post-fuzz `SELECT 1` probe is rejected: `Code: 241 ... (total) memory limit exceeded: would use 55.80 GiB (attempt to allocate chunk of 0.00 B), current RSS: 57.11 GiB, maximum: 46.32 GiB`.
- With zero queries running, tracked usage and RSS keep climbing (~110 MB/s): RSS 55.39 -> 57.71 GiB over the 21 s probe window. The cap is `max_server_memory_usage_to_ram_ratio = 0.75` of a 61.76 GiB runner.
- ~23 s into the probe loop the job is canceled (`The operation was canceled.`), before `status.tsv` is written and before any artifact upload. No server.log, no fuzzer.log, no CIDB row: the offending query history is lost, so the memory growth cannot be attributed.
The harness contributes on two points (`ci/jobs/scripts/fuzzer/run-fuzzer.sh`):
1. The post-fuzz liveness loop treats `MEMORY_LIMIT_EXCEEDED` as "alive, give it time to reclaim" and retries up to ~2 min. It cannot distinguish a transient post-query spike from tracked memory pinned above the cap while idle.
2. If the loop exits with `server_died=0`, the script attempts a graceful `clickhouse stop` with an unbounded `wait`. Graceful shutdown of a memory-saturated debug server is unreliable, and RSS keeps growing toward physical RAM, so the job hangs until something kills it (workflow cancel, runner death, or kernel OOM), destroying the evidence.
### Frequency (CIDB, 2026-07-25)
Vanished-job runs (one targeted twin has CIDB rows for a SHA, the other has none because the job died before reporting), last 7 days: 4 missing `old_compatibility` + 6 missing plain `targeted` - about 1/day per variant. The mode is symmetric across variants (the harness path is shared by all AST fuzzer and BuzzHouse variants), not specific to `old_compatibility`.
```sql
SELECT pull_request_number, left(commit_sha,12) sha, min(check_start_time) t
FROM default.checks
WHERE check_name = 'AST fuzzer (amd_debug, targeted)'
AND check_start_time > now() - INTERVAL 7 DAY
AND commit_sha NOT IN (
SELECT DISTINCT commit_sha FROM default.checks
WHERE check_name = 'AST fuzzer (amd_debug, targeted, old_compatibility)'
AND check_start_time > now() - INTERVAL 8 DAY)
GROUP BY 1,2 ORDER BY t DESC
```
(and the symmetric query with the two check names swapped)
Note the decomposition: the bulk of `old_compatibility` FAIL rows across unrelated PRs (28 rows / 24 distinct failure names in the trailing 3 days as of filing; ~90 rows in the 3-day window around 2026-07-24) are attributed per-query crashes with distinct STIDs (count 1-5 each) - pre-existing trunk bugs surfaced by fuzzing, individually actionable per STID, not one shared bug. This issue tracks only the unattributable idle-memory mode above.
### Harness fix
Fix PR: link to follow shortly.
It bounds every post-budget stage of `run-fuzzer.sh` (client reap, liveness probing, teardown) and makes the stuck state a first-class failure: the probe loop detects persistent global-tracker rejection (two tiers: 12 consecutive rejections with host `MemAvailable` under 4 GiB, or 60 consecutive regardless), records a `server_memory_stuck.txt` marker with the tracker line, SIGKILLs the server, and lets the job complete its normal artifact upload with a deterministic FAIL row (`Server unresponsive: memory limit exceeded`). Reap/teardown escalations write a `harness_watchdog.txt` and surface as ERROR. Fuzz budgets and the client kill grace are unchanged; fail fast with evidence instead of hanging until the box dies.
### Open follow-up (kept open after the harness fix merges)
Root-cause the idle post-fuzz memory growth from the artifacts the fix makes available (server.log MemoryTracker lines + fuzzer.log query history of stuck runs). Candidates: accumulated fuzzed Memory/Set/Join table state, background merges/mutations of fuzzed tables (merge memory is not bounded by query limits, see #107410), system-log flush feedback under memory pressure.
Related: #109590 (a distinct sub-mode: fuzzer client exits 241 while the server stays healthy - job-classification fix only).
Contributor guide
Research direction
Start with ci/jobs/scripts/fuzzer/run-fuzzer.sh and the artifacts produced by the harness fix, especially server.log and fuzzer.log from stuck runs. Compare MemoryTracker lines with the recorded query history, then investigate the stated candidates: accumulated fuzzed table state, background merges or mutations, and system-log flushing. Done means identifying and fixing the idle post-fuzz memory growth, with a reproducible test or evidence from affected runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, shell, sql
- Domain
- ci-cd, databases, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100