IntersectMBO / IntersectMBO/ouroboros-consensus

Use `posix_fadvise` on ImmutableDB chunk files to bound page cache footprint during sync and replay

Open
#1,999 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
67
Forks
43
Avg merge
5d 13h
Merged PRs (30d)
43

Description

## Summary

During both sync and replay, ImmutableDB chunk files are read and written sequentially without giving the kernel any hint about the access pattern. On Linux this leaves everything in the page cache indefinitely, even though chunks have effectively single-use access semantics from the node's perspective (no rollbacks cross into the immutable region; block-serving to peers only matters near tip). On memory-constrained hosts this consumes several GB of RAM that is structurally useless to the node.

This issue proposes adding a `posix_fadvise`/`madvise` hook to the filesystem abstraction and using it on chunk open/close and on chunk completion during append.

## Observed behaviour

`cardano-node` syncing mainnet from genesis on a Raspberry Pi 5 (8 GB RAM, NVMe SSD, no other meaningful load):

- During Byron replay, `/proc/meminfo: Cached` grows almost perfectly linearly with `ChainDB.tip.slot`, from ~470 MB at startup to ~2.7 GB after ~2 h 30 m and ~2.3 M slots.
- Slope ≈ 1.3 KB / slot, matching the expected Byron per-block ImmutableDB footprint.
- `vmtouch -v db/immutable/` confirms the cache is dominated by immutable chunk files. The LSM ledger state on disk contributes negligibly — as expected, since Byron replay has not yet materialised significant UTxO-HD state.

This was observed during a cold sync (writing chunks as blocks arrive from upstream), not a pure replay pass, so the pattern applies to the write path as well as the read path.

Why it matters:

- `Cached` is reclaimable, so this is not a correctness issue and `MemAvailable` stays healthy during Byron.
- But the trajectory extrapolates. Post-Byron blocks are larger, and the LSM backend will start carrying non-trivial on-disk state that *does* benefit from being cached. Giving ~hundreds of MB to use-once chunk data competes with that legitimately-hot LSM state for RAM, and with the RTS heap, on 8 GB targets.
- The optimisation is architecturally clean on any host: we know these pages are single-use; their retention has no functional benefit to the node.

## Background: what `posix_fadvise` offers

`posix_fadvise(fd, offset, len, advice)` lets the process tell the kernel how a file range will be accessed. The relevant advices:

- `POSIX_FADV_SEQUENTIAL` — increases readahead for streaming reads.
- `POSIX_FADV_DONTNEED` — drops the specified range from the page cache.
- `POSIX_FADV_WILLNEED` — prefetches a range.
- `POSIX_FADV_NOREUSE` — "accessed once"; historically a no-op on Linux, honoured on 6.3+.

A codebase search for `fadvise`, `madvise`, `mmap`, `O_DIRECT` finds no existing usage anywhere in `ouroboros-consensus`.

## Proposal

### 1. `fs-api` (upstream) — extend `HasFS`

Add an `hAdvise` operation with a no-op default for non-POSIX backends (`IOSim`, Windows for the first cut):

```haskell
data Advice = AdviceNormal | AdviceSequential | AdviceRandom
| AdviceWillNeed | AdviceDontNeed | AdviceNoReuse

hAdvise :: HasFS m h -> Handle h -> Word64 {- offset -} -> Word64 {- length, 0 = whole file -} -> Advice -> m ()
```

Linux/POSIX backend wraps `posix_fadvise(2)`; Windows backend is a no-op for now (can later be backed by `SetFileInformationByHandle` / `FILE_FLAG_SEQUENTIAL_SCAN`).

### 2. ImmutableDB read path

In `ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Iterator.hs`:

- In `iteratorStateForChunk` (currently around L537–545), immediately after opening a chunk file with `hOpen … ReadMode`, call `hAdvise handle 0 0 AdviceSequential`.
- When the iterator transitions past a chunk (closing the handle), call `hAdvise handle 0 0 AdviceDontNeed` before `hClose` so those pages are evicted.

### 3. ImmutableDB write path

When appending, each completed chunk is written sequentially, fsynced, and then (once the tip advances into the next chunk) never written again. Apply `AdviceDontNeed` to the completed chunk after its final `hSync`, in the appender module (exact call site needs a short investigation — likely the code path that rolls over to a new chunk in `Impl/State.hs` / its appender helper). During active append of the current chunk the hint should NOT be used, since we do read back freshly-written blocks during ledger application.

## Open questions

- **Block serving at tip.** Once the node is caught up, BlockFetch requests from peers benefit from cached chunks near tip. `DontNeed` is always safe for chunks older than `k` blocks behind the immutable tip; the simplest policy is to apply it only to chunks that have been fully consumed and are not near tip. During sync-from-genesis this distinction is moot — no meaningful peer traffic is being served.
- **Granularity.** Per-chunk at close time is simpler and almost certainly sufficient; per-read-chunk streaming hints are overkill.
- **Ordering vs. `fSync`.** On the write path, `AdviceDontNeed` only drops clean pages; we must ensure writeback has completed (explicit `hSync` or `sync_file_range` with `WAIT_AFTER`) before advising, otherwise the hint is silently ineffective for dirty pages.
- **Upstream coordination.** The `HasFS` change lives in `fs-api` (separate repo). We may want to land the consensus-side call sites behind a CPP guard or a conditional dependency bump.
- **IOSim semantics.** No-op is the right default. We don't need to model cache behaviour in simulation.

## Acceptance

- `vmtouch -v db/immutable/` during a cold sync shows page cache residency bounded by the number of chunks held open by active iterators + the currently-appended chunk, rather than growing linearly with tip slot.
- No regression in sync throughput. On NVMe the re-read penalty if a hint turned out to be wrong is negligible; on slower storage, this matters only if we incorrectly `DontNeed` a range we will re-read, which the policy above avoids.
- Unit/integration tests for the new `HasFS` operation in `fs-api` (no-op semantics) and a consensus-level test that exercises the iterator close path.

## Motivating graph from the Raspberry Pi

Image

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the HasFS definition in fs-api and the proposed call sites in ouroboros-consensus/.../ImmutableDB/Impl/Iterator.hs and Impl/State.hs. Trace chunk open, iterator close, rollover, and final hSync paths before checking the fs-api no-op tests and consensus iterator-close coverage. Done means bounded immutable-chunk cache residency, no sync-throughput regression, and passing unit/integration tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell, linux
Domain
blockchain, databases, operating-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.