IntersectMBO / IntersectMBO/cardano-node-tests

Node BUG: Mainnet sync: peak RAM nearly doubled in node 11.1.0 (15.4 -> 36.2 GiB)

Open
#3,669 0 comments 0 reactions 0 assignees View on GitHub
node upstream bug
Dominant language
Python
Stars
59
Forks
28
Avg merge
2h 30m
Merged PRs (30d)
62

Description

## Summary

Full mainnet sync from genesis with node **11.1.0** consumed roughly twice the memory of
**11.0.1** while getting faster. Peak RSS went from 15.43 GiB to 36.15 GiB (+134%), and
peak retained live data from 10.28 GiB to 25.65 GiB (+150%).

Full report, including per-era timings and graphs:
https://docs.google.com/document/d/e/2PACX-1vTujvaT-knCY4ZTUVqUelUIww8n9LJhbd8ZnBjMsUxAp4fvpfmcVFux-qENSzL057vXPxON5LC8_UMk/pub

## What was measured

Full mainnet syncs from genesis on the same host (32 cores, 134 GB RAM, `V2InMemory`
LedgerDB backend, ghc-9.6). Memory figures are parsed from the node's own `Resources:`
trace lines, so `GC live bytes` is what the collector could not reclaim, as distinct from
heap the RTS happens to be holding.

| | 11.0.1 | 11.1.0 | change |
|---|---|---|---|
| Peak RSS | 15.43 GiB | **36.15 GiB** | +134% |
| Mean RSS | 12.95 GiB | 18.92 GiB | +46% |
| Peak GC live bytes | 10.28 GiB | **25.65 GiB** | +150% |
| Mean GC live bytes | 6.03 GiB | 9.06 GiB | +50% |
| Major GCs | 1,896 | 1,331 | -30% |
| Total allocation | 336 TB | 285 TB | -15% |
| Sync time | 133,009 s | 116,618 s | -12.3% |
| Mean CPU load | 152% | 156% | +2.6% |

Speed was not affected: 11.1.0 finished 4 h 33 min faster and advanced 6.9 M slots
further, with CPU load flat. The regression is purely in memory.

No OOM was recorded: the test host has 134 GB, so a 36 GiB peak is survivable there but
not on the 16-32 GB machines many operators run.

## Shape of the regression

The two releases produce distinct heap profiles, and the shape is what points at the
cause:

- **11.0.1** - monotone staircase. Climbs in steps to a 15.34 GiB plateau at slot 134 M
and never gives memory back: 21 heap decreases across 130,674 samples, nothing above
20 GiB.
- **11.1.0** - volatile sawtooth. Oscillates roughly 19-36 GiB with 204 heap decreases.
34.7% of samples sit above 20 GiB and 6.0% above 30 GiB across 21 separate excursions.
The pattern starts early: 11.1.0 is already at 9.16 GiB RSS before slot 20 M, where
11.0.1 is at 2.24 GiB.

Three details indicate retention rather than allocation churn:

1. The regression is in `GC live bytes`, i.e. data the collector could not reclaim, not
just in heap the RTS was holding spare.
2. Node 11.1.0 ran **fewer** major GCs than 11.0.1 (1,331 vs 1,896) despite holding far
more heap, so the heap was ballooning between collections.
3. It allocated **less** in total than 11.0.1 (285 vs 336 TB) while retaining much more,
so this is not extra garbage being produced.

The run took 386 snapshots at a median interval of 5.0 minutes, and the RSS excursions
recur on a comparable cadence - consistent with ledger states accumulating until each
snapshot completed and only then being collected.

## Root cause

[ouroboros-consensus#2227](https://github.com/IntersectMBO/ouroboros-consensus/pull/2227),
"Decouple LedgerDB garbage collection from snapshotting", merged 2026-08-27 onto
`ouroboros-consensus-4.x-backports`:

> As the thread delay for snapshotting was blocking the snapshotting thread, we would
> accumulate states in the LedgerDB until the snapshot was taken.

The delay now runs in its own thread, so LedgerDB garbage collection is no longer gated
on snapshotting.

Contributor guide

Open the contributing guide

Research direction

Start with the reported 11.0.1 versus 11.1.0 measurements and the linked ouroboros-consensus#2227 pull request, which identifies LedgerDB garbage collection being gated on snapshotting. Compare the relevant backport status and reproduce or validate the sync memory profile; done means confirming whether the backport removes the observed retention regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.