IntersectMBO / IntersectMBO/cardano-node-tests
Node BUG: Mainnet sync: node 11.1.1 still retains ~55% more live data than 11.0.1
- Dominant language
- Python
- Stars
- 59
- Forks
- 28
- Avg merge
- 2h 30m
- Merged PRs (30d)
- 62
Description
## Summary
Node **11.1.1** resolved the memory regression reported in #3669, but full mainnet sync
still uses noticeably more memory than the **11.0.1** baseline:
- Peak RSS **+37%** (15.43 -> 21.16 GiB)
- Peak GC live bytes **+55%** (10.28 -> 15.94 GiB)
This is not the same fault as #3669. That one was an oscillation - heap swinging between
19 and 36 GiB because LedgerDB garbage collection was gated on snapshotting, fixed by
[ouroboros-consensus#2227](https://github.com/IntersectMBO/ouroboros-consensus/pull/2227).
Node 11.1.1 no longer oscillates; it sits on a *higher plateau* than 11.0.1 did, which
suggests a different cause rather than an incomplete fix.
Under investigation. It is not yet clear whether this is expected for the V2 LedgerDB
backend or a genuine second regression, and there is no consensus fix for it yet.
Full report, including per-era timings and graphs:
https://tests.cardano.intersectmbo.org/test_results/sync_reports/mainnet_11_1_1.html
## Measurements
Full mainnet syncs from genesis on the same host (32 cores, 134 GB RAM, `V2InMemory`
LedgerDB backend, ghc-9.6). Memory is parsed from the node's own `Resources:` trace
lines, so `GC live bytes` is what the collector could not reclaim, as distinct from heap
the RTS happens to be holding.
| | 11.0.1 | 11.1.1 | change |
|---|---|---|---|
| Peak RSS | 15.43 GiB | **21.16 GiB** | **+37%** |
| Mean RSS | 12.95 GiB | 15.67 GiB | +21% |
| Peak GC live bytes | 10.28 GiB | **15.94 GiB** | **+55%** |
| Mean GC live bytes | 6.03 GiB | 6.86 GiB | +14% |
| Major GCs | 1,896 | 1,940 | +2% |
| Total allocation | 336 TB | 286 TB | -15% |
| Sync time | 133,009 s | 110,606 s | -16.8% |
| Mean CPU load | 152% | 157% | +3.3% |
Node revision under test: `c2ebdc87dfe07706a83e52f219e712c60d1b0a56`.
Speed is not affected and in fact improved by 6 h 13 min. Allocation went *down* while
retention went up, so this is not extra garbage being produced. Major GC count is normal,
so the collector is running as often as it did on 11.0.1 - it simply has more it cannot
free.
## The heap profile is healthy, just higher
- **11.0.1** - monotone staircase to a 15.34 GiB plateau reached at slot 134 M. 21 heap
decreases across 130,674 samples, nothing above 20 GiB.
- **11.1.1** - also a staircase, plateauing near 19-20 GiB. 90 heap decreases, 3.5% of
samples above 20 GiB, **none above 25 GiB**.
For contrast, 11.1.0 had 204 heap decreases with 34.7% of samples above 20 GiB and 6.0%
above 30 GiB. The sawtooth is gone.
## It is not chain growth
The 11.1.1 run covered 8.2 M more slots and 2.8 GB more chain than the 11.0.1 run, which
is far too little to explain the gap:
- The divergence starts around **slot 40 M** and widens monotonically from there.
- Truncating all runs to a common window of slots <= 188 M leaves the peaks unchanged:
15.43 GiB for 11.0.1 against 21.16 GiB for 11.1.1.
- Node 11.1.1 crosses the 15.34 GiB level that 11.0.1 uses as its ceiling at **slot 71 M**,
roughly halfway through Alonzo, with more than half the chain still to sync.
Peak RSS by slot band, GiB:
| Slot band | 11.0.1 | 11.1.1 |
|---|---|---|
| 0-10 M | 2.24 | 2.13 |
| 20-30 M | 4.09 | 4.84 |
| 40-50 M | 8.26 | 9.69 |
| 60-70 M | 11.49 | 15.33 |
| 80-90 M | 13.55 | 18.12 |
| 100-110 M | 14.15 | 19.01 |
| 120-130 M | 15.43 | 21.16 |
| 150-160 M | 15.43 | 20.36 |
| 180-190 M | 15.43 | 20.36 |
Early sync is at parity - 11.1.1 is actually marginally lower through the Byron era - and
the two curves separate from Alonzo onward.
## Why it matters
A 21 GiB peak is comfortable on the 134 GB test host, and no OOM occurred. It is less
comfortable for operators on 16-32 GB machines, which is the constituency the 11.0.1
figure of 15.4 GiB was already close to.
Contributor guide
Research direction
Start with the linked full sync report and compare the 11.0.1 and 11.1.1 runs at revision c2ebdc87dfe07706a83e52f219e712c60d1b0a56, focusing on the divergence from Alonzo onward. Done means determining whether the higher V2InMemory LedgerDB retention is expected or a regression, and identifying a consensus fix or clear evidence for the conclusion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100