node: fetched-block cache fills during finality stall and never drains — node permanently stuck after ~1024 slots without finalization
- Dominant language
- Zig
- Stars
- 97
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
When finality stalls on the network for a prolonged period (observed: ~8 hours on devnet-4), a zeam node's fetched-block cache silently fills to its 1024-entry cap and then **permanently** rejects all new blocks, including the ones it would need to catch up. There is no self-recovery: the only way out is restarting with `--checkpoint-sync-url`.
Because the cache is **pruned only on finalization events**, any stall longer than `MAX_CACHED_BLOCKS` slots (~68 min at 4s/slot) is sufficient to wedge the node. If that node is the configured aggregator, the wedge removes aggregations from the network and can amplify the original stall.
## Observed behavior (devnet-4, 2026-04-17)
- All 8 clients across the network reported `finalized=3797, justified=3802` and stopped advancing. (Initial cause of the stall is not the subject of this issue — see caveat below.)
- Non-zeam clients continued to produce blocks; network head advanced to ~slot 12085.
- zeam's head was frozen at slot 3806. Its logs filled with:
```
[warning] Cache full (1024 blocks), rejecting future block 0x.. at slot 12078
[warning] Cache full (1024 blocks), rejecting block 0x.. at slot 12080
[error] error.UnknownSourceBlock producing aggregation for slot=3806
```
- curl `/lean/v0/checkpoints/justified` on zeam returned `slot=3800` while peers returned `slot=3802`.
- Recovery required \`spin-node.sh --restart-client zeam_0 --checkpoint-sync-url https://leanpoint.leanroadmap.org/lean/v0/states/finalized\`.
## Root cause analysis
Cache admission in \`pkgs/node/src/node.zig\`:
\`\`\`zig
// If cache is full, reject - proactive pruning on finalization keeps the cache bounded
if (self.network.fetched_blocks.count() >= constants.MAX_CACHED_BLOCKS) {
self.logger.warn(\"Cache full ({d} blocks), rejecting block ...\", ...);
return CacheBlockError.CachingFailed;
}
\`\`\`
(also mirrored in \`cacheFutureBlock\` at ~line 593)
Pruning is wired via \`prune_cached_blocks_fn\` and invoked only when finalization advances (\`chain.zig:1102\`). \`constants.zig\` defines:
\`\`\`zig
pub const MAX_CACHED_BLOCKS = 1024;
\`\`\`
The failure mode is a feedback loop:
1. Something causes finalization to stop advancing (consensus issue, network partition, proposer outage — outside this issue).
2. The cache stops being pruned and grows by ~1 entry per gossiped / future block.
3. After ~1024 slots without finalization, the cache is full.
4. From that point on every new block is rejected, including the ones the node would fetch to resolve the stall (sync requests, parent backfills).
5. The node becomes permanently out of sync; even if the rest of the network resumes finalizing, this node cannot catch up because it cannot ingest the blocks it needs.
The code comment *\"proactive pruning on finalization keeps the cache bounded\"* is only accurate in the happy path. It is precisely the pathological case (no finalization) where unbounded growth would happen, and the current mitigation (hard reject) converts a transient liveness issue into a permanent one.
## Suggested directions (non-prescriptive)
Any of these would break the feedback loop:
1. **Evict by slot, not reject.** When \`fetched_blocks.count() >= MAX_CACHED_BLOCKS\`, drop the oldest (or highest-slot orphan with no descendants) instead of refusing ingress.
2. **Time-based pruning independent of finalization.** Periodically evict entries older than N slots regardless of whether finality has advanced.
3. **Distinguish sync-response blocks from gossip.** Blocks fetched in response to our own \`blocks_by_root\` requests should not be refused — they are the recovery path. Consider a separate admission policy (or a separate cache) for in-flight sync responses.
4. **Trigger checkpoint-sync fallback automatically.** When the node has been \`behind_peers\` for more than X slots and the cache is full, attempt a forced re-sync from a configured endpoint instead of waiting for human intervention.
Option 3 alone would likely have been sufficient to recover this incident without any operator action.
## Reproduction
Any scenario producing a finality gap of >\`MAX_CACHED_BLOCKS\` slots while the node remains connected to peers that keep gossiping new blocks. Easiest synthetic repro: lower \`MAX_CACHED_BLOCKS\` to 64 in a devnet, take down >1/3 of validators for ~5 minutes, observe zeam nodes fail to recover even after the validators return.
## References
- \`pkgs/node/src/node.zig\` L537-545 (cacheBlock admission)
- \`pkgs/node/src/node.zig\` L593-600 (cacheFutureBlock admission)
- \`pkgs/node/src/constants.zig\` L16-18 (\`MAX_CACHED_BLOCKS = 1024\`)
- \`pkgs/node/src/chain.zig\` L1102-1106 (pruning invocation, finalization-gated)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with cacheBlock and cacheFutureBlock in pkgs/node/src/node.zig, then trace pruning from chain.zig and the MAX_CACHED_BLOCKS definition in constants.zig. Reproduce with a reduced cache limit and a prolonged finality gap. Done means the node can ingest the blocks needed to recover after finalization resumes, without the cache remaining permanently full.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- zig
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100