FluffyLabs / FluffyLabs/typeberry
Block pruning: values DB leaks memory (ref-counting not implemented)
- Dominant language
- TypeScript
- Stars
- 10
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
During block pruning (introduced in #930), `BlocksDb` and `StatesDb` are pruned as a sliding window, but the **values DB is never cleared**, causing a memory leak over long-running fuzzer sessions.
## Root cause
In `packages/jam/database/serialized-states-db.ts`, `InMemorySerializedStates` holds a shared `valuesDb`:
```ts
// packages/jam/database/serialized-states-db.ts
private readonly valuesDb: HashDictionary = HashDictionary.new();
```
Values (large leaf node payloads) are inserted into this map on every state update:
```ts
for (const val of values) {
this.valuesDb.set(val[0], val[1]);
}
```
However, when a state is pruned via `markUnused`, entries in `valuesDb` are **never removed**. The map grows unboundedly.
## Why it's hard to fix
Removing a value is only safe when **no remaining leaf node in any retained state** still references it. This requires ref-counting across states. There is already a related TODO in the leaf update logic:
```ts
// packages/jam/database/leaf-db-update.ts (line 30)
// TODO [ToDr] Handle ref-counting values or updating some header-hash-based references.
```
## Suggested fix
Implement ref-counting in `valuesDb`:
- Increment a ref-count when a value hash is written to a new leaf node.
- Decrement the ref-count (and delete the entry) when a leaf node is removed or when a state that references the value is pruned.
## Impact
The leak is bounded in practice by the rate of new large values being written, but for long-running fuzzer sessions it can accumulate significantly.
---
_Reported from PR #930 (https://github.com/FluffyLabs/typeberry/pull/930). Requested by @tomusdrw._
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in packages/jam/database/serialized-states-db.ts, tracing valuesDb insertion and state pruning through markUnused. Then read the ref-counting TODO in packages/jam/database/leaf-db-update.ts. Done means values referenced by retained states remain available while entries no longer referenced after pruning are removed, preventing unbounded growth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100