IntersectMBO / IntersectMBO/cardano-node-tests
Node BUG: Mainnet sync: peak RAM nearly doubled in node 11.1.0 (15.4 -> 36.2 GiB)
- Dominant language
- Python
- Stars
- 59
- Forks
- 28
- Avg merge
- 2h 30m
- Merged PRs (30d)
- 62
Description
## Summary
Full mainnet sync from genesis with node **11.1.0** consumed roughly twice the memory of
**11.0.1** while getting faster. Peak RSS went from 15.43 GiB to 36.15 GiB (+134%), and
peak retained live data from 10.28 GiB to 25.65 GiB (+150%).
Full report, including per-era timings and graphs:
https://docs.google.com/document/d/e/2PACX-1vTujvaT-knCY4ZTUVqUelUIww8n9LJhbd8ZnBjMsUxAp4fvpfmcVFux-qENSzL057vXPxON5LC8_UMk/pub
## What was measured
Full mainnet syncs from genesis on the same host (32 cores, 134 GB RAM, `V2InMemory`
LedgerDB backend, ghc-9.6). Memory figures are parsed from the node's own `Resources:`
trace lines, so `GC live bytes` is what the collector could not reclaim, as distinct from
heap the RTS happens to be holding.
| | 11.0.1 | 11.1.0 | change |
|---|---|---|---|
| Peak RSS | 15.43 GiB | **36.15 GiB** | +134% |
| Mean RSS | 12.95 GiB | 18.92 GiB | +46% |
| Peak GC live bytes | 10.28 GiB | **25.65 GiB** | +150% |
| Mean GC live bytes | 6.03 GiB | 9.06 GiB | +50% |
| Major GCs | 1,896 | 1,331 | -30% |
| Total allocation | 336 TB | 285 TB | -15% |
| Sync time | 133,009 s | 116,618 s | -12.3% |
| Mean CPU load | 152% | 156% | +2.6% |
Speed was not affected: 11.1.0 finished 4 h 33 min faster and advanced 6.9 M slots
further, with CPU load flat. The regression is purely in memory.
No OOM was recorded: the test host has 134 GB, so a 36 GiB peak is survivable there but
not on the 16-32 GB machines many operators run.
## Shape of the regression
The two releases produce distinct heap profiles, and the shape is what points at the
cause:
- **11.0.1** - monotone staircase. Climbs in steps to a 15.34 GiB plateau at slot 134 M
and never gives memory back: 21 heap decreases across 130,674 samples, nothing above
20 GiB.
- **11.1.0** - volatile sawtooth. Oscillates roughly 19-36 GiB with 204 heap decreases.
34.7% of samples sit above 20 GiB and 6.0% above 30 GiB across 21 separate excursions.
The pattern starts early: 11.1.0 is already at 9.16 GiB RSS before slot 20 M, where
11.0.1 is at 2.24 GiB.
Three details indicate retention rather than allocation churn:
1. The regression is in `GC live bytes`, i.e. data the collector could not reclaim, not
just in heap the RTS was holding spare.
2. Node 11.1.0 ran **fewer** major GCs than 11.0.1 (1,331 vs 1,896) despite holding far
more heap, so the heap was ballooning between collections.
3. It allocated **less** in total than 11.0.1 (285 vs 336 TB) while retaining much more,
so this is not extra garbage being produced.
The run took 386 snapshots at a median interval of 5.0 minutes, and the RSS excursions
recur on a comparable cadence - consistent with ledger states accumulating until each
snapshot completed and only then being collected.
## Root cause
[ouroboros-consensus#2227](https://github.com/IntersectMBO/ouroboros-consensus/pull/2227),
"Decouple LedgerDB garbage collection from snapshotting", merged 2026-08-27 onto
`ouroboros-consensus-4.x-backports`:
> As the thread delay for snapshotting was blocking the snapshotting thread, we would
> accumulate states in the LedgerDB until the snapshot was taken.
The delay now runs in its own thread, so LedgerDB garbage collection is no longer gated
on snapshotting.
Contributor guide
Research direction
Start with the reported 11.0.1 versus 11.1.0 measurements and the linked ouroboros-consensus#2227 pull request, which identifies LedgerDB garbage collection being gated on snapshotting. Compare the relevant backport status and reproduce or validate the sync memory profile; done means confirming whether the backport removes the observed retention regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100