FluffyLabs / FluffyLabs/typeberry

Block pruning: values DB leaks memory (ref-counting not implemented)

Open
#931 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary

During block pruning (introduced in #930), `BlocksDb` and `StatesDb` are pruned as a sliding window, but the **values DB is never cleared**, causing a memory leak over long-running fuzzer sessions.

## Root cause

In `packages/jam/database/serialized-states-db.ts`, `InMemorySerializedStates` holds a shared `valuesDb`:

```ts
// packages/jam/database/serialized-states-db.ts
private readonly valuesDb: HashDictionary = HashDictionary.new();
```

Values (large leaf node payloads) are inserted into this map on every state update:

```ts
for (const val of values) {
this.valuesDb.set(val[0], val[1]);
}
```

However, when a state is pruned via `markUnused`, entries in `valuesDb` are **never removed**. The map grows unboundedly.

## Why it's hard to fix

Removing a value is only safe when **no remaining leaf node in any retained state** still references it. This requires ref-counting across states. There is already a related TODO in the leaf update logic:

```ts
// packages/jam/database/leaf-db-update.ts (line 30)
// TODO [ToDr] Handle ref-counting values or updating some header-hash-based references.
```

## Suggested fix

Implement ref-counting in `valuesDb`:
- Increment a ref-count when a value hash is written to a new leaf node.
- Decrement the ref-count (and delete the entry) when a leaf node is removed or when a state that references the value is pruned.

## Impact

The leak is bounded in practice by the rate of new large values being written, but for long-running fuzzer sessions it can accumulate significantly.

---

_Reported from PR #930 (https://github.com/FluffyLabs/typeberry/pull/930). Requested by @tomusdrw._

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in packages/jam/database/serialized-states-db.ts, tracing valuesDb insertion and state pruning through markUnused. Then read the ref-counting TODO in packages/jam/database/leaf-db-update.ts. Done means values referenced by retained states remain available while entries no longer referenced after pruning are removed, preventing unbounded growth.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.