Unbounded memory growth wedges the event loop via cgroup reclaim throttling — no OOM kill, so the container never restarts

## Summary

A long-running instance grew its working set until the container's cgroup swap allowance was
100% consumed. At that point the kernel began throttling the process on every allocation via
`memory.high` reclaim rather than OOM-killing it, which starved the Node event loop for minutes
at a time.

The critical property is that **nothing recovered and nothing alarmed**. Because the process was
throttled instead of killed, `oom_kill` stayed at `0`, the container's `restart=always` policy
never fired, and Docker reported the container healthy and `Up 4 weeks` for the entire
degradation. The only signal that eventually surfaced was an external synthetic HTTP probe, hours
after the instance had started stalling.

Observed on `harper-pro:5.1.18` after 33 days of process uptime, in a container limited to
4 GiB memory + 4 GiB swap.

## Evidence

cgroup v2 values read from `/sys/fs/cgroup` at time of investigation:

| Metric | Value |
|---|---|
| `memory.high` | `3221225472` (75% of `memory.max`) |
| `memory.current` | `3220811776` — pinned at `memory.high` |
| `memory.swap.current` | `4294856704` / `4294967296` max — **99.997%** |
| `memory.events` → `high` | **35589175** |
| `memory.events` → `oom` / `oom_kill` | `0` / `0` |

`ps` inside the container showed PID 1 at ~2.93 GiB RSS. Total anonymous demand (RSS + swapped)
was roughly 7.4 GB against a 4 GiB cap.

## Symptoms this produced

Harper's own analytics reported main-thread task-queue latency escalating through the day:

```
07:09 13,126 ms
08:19 357,835 ms
09:23 393,388 ms
11:33 106,264 ms
13:03 1,015,492 ms
14:00 1,687,859 ms
```

(These values exceed the interval between consecutive log lines, so the metric reflects
accumulated age of queued tasks rather than one uninterrupted blocked interval. The trend is the
point.)

Accompanied throughout by:

- `JavaScript execution has taken too long and is not allowing proper event queue cycling` on
`main/0`, `http/3` and `http/4`
- Replication churn against a healthy peer — 1006 disconnects, receive-watchdog terminations,
`Client network socket disconnected before secure TLS connection was established`, and repeated
wedged-subscription reconciliation
- `Transaction was open too long and has been committed` on `hdb_license/` and `hdb_analytics/`,
plus a >1 min read transaction on a worker
- `docker exec` into the container hanging past 180s, while the same command against a sibling
container on the same host returned instantly
- HTTP intermittently serving normally (200 in ~12 ms during a lucid window) and not responding at
all during a stall — which is what makes a synthetic probe flap rather than fail cleanly

## Ruled out during investigation

Recording these because they are the loudest lines in the log and read as causes:

- **The replication peer.** Probed directly: replication port open, HTTPS 200 in 0.295 s. The
peer is healthy; the connection failures originate locally because the event loop cannot
service the sockets.
- **Host memory pressure / noisy neighbours — as the cause.** The throttling is driven by this
container's own `memory.high` (`3221225472`, 75% of its `memory.max`), with `memory.current`
pinned essentially exactly on that watermark. `memory.high` compares a cgroup's usage against its
own limit, so this would occur identically on an otherwise idle host; host free RAM does not
enter into it.

Stated precisely, because the host is not blameless: the host **is** broadly oversubscribed —
all 70 containers were swapping, ~40 GB combined, with under 1 GB of RAM free. This container was
the largest single consumer (~4 GB) and the only one to have exhausted its own swap ceiling, but
it accounted for only ~10% of host swap. Host contention makes each fault more expensive and very
plausibly deepens the stalls; it does not create them.
- **A crash or restart loop.** Process uptime was continuous at 33 days.

## Open question — where the growth comes from

This is unattributed and is the main thing the issue needs. The instance runs several
application components, and a scheduled reporting job correlates with the sharp inflection in
stall frequency, but low-level stall warnings predate it by several hours. So this could be:

1. A leak or unbounded retention in engine code, or
2. A component pulling a large result set into memory, with the 4 GiB cap simply being too small
for the workload

Distinguishing these needs a heap snapshot taken before the instance is recycled. **That snapshot
was not captured, and the instance has since been restarted to restore service (2026-08-23) — so
the live evidence for this occurrence is gone.** Under full reclaim throttling a ~3 GB heap
snapshot risked deepening the stall, and restoring service was chosen over preserving it.

Post-restart the condition cleared completely: `memory.current` 3.29 GB → 755 MB,
`memory.swap.current` 4.29 GB → 0, `memory.events.high` 116,988,511 → 0, and memory PSI
`full avg60` 76.98% → 0.00. That confirms the mechanism but leaves attribution open. **Whoever
picks this up should take a heap snapshot on the next occurrence, before recycling** — growth was
monotonic over 33 days, so if it is a leak it should recur and be observable well before it
saturates again.

## Why this is worth fixing regardless of attribution

Even if the growth turns out to be workload-shaped rather than a leak, the *failure mode* is bad
independently of its cause:

- Throttling instead of OOM-killing means the container-level safety net (`restart=always`) never
engages. The instance degrades indefinitely rather than restarting.
- Nothing in Harper's own status output or in container health reflects the condition. An operator
looking at `docker ps` sees a healthy container.
- A cgroup-level signal (`memory.swap.current` approaching `memory.swap.max`, or a rising
`memory.events.high` rate) would have surfaced this hours earlier and would catch the same
failure mode on any instance, not just this one.

Related in theme, though a different mechanism: #696 (process stays alive with no listeners and
no container restart).

## Environment

- `harper-pro:5.1.18` (current release line at time of filing: 5.2.4)
- Container limits: `Memory=4294967296`, `MemorySwap=8589934592`, `RestartPolicy=always`
- cgroup v2, Linux host
- Process uptime at observation: 33d 15h

Start with the cgroup v2 metrics under /sys/fs/cgroup and the event-loop stall warnings, then monitor the next recurrence before recycling the instance. Capture a heap snapshot while growth is observable and compare it with the scheduled reporting job to determine whether the cause is retention or a large result set; done requires attribution and a defined mitigation for the throttling failure mode.

Written by the indexing model from the issue text.