blockblaz / blockblaz/zeam

node: fetched-block cache fills during finality stall and never drains — node permanently stuck after ~1024 slots without finalization

Open
#749 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Zig
Stars
97
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Summary

When finality stalls on the network for a prolonged period (observed: ~8 hours on devnet-4), a zeam node's fetched-block cache silently fills to its 1024-entry cap and then **permanently** rejects all new blocks, including the ones it would need to catch up. There is no self-recovery: the only way out is restarting with `--checkpoint-sync-url`.

Because the cache is **pruned only on finalization events**, any stall longer than `MAX_CACHED_BLOCKS` slots (~68 min at 4s/slot) is sufficient to wedge the node. If that node is the configured aggregator, the wedge removes aggregations from the network and can amplify the original stall.

## Observed behavior (devnet-4, 2026-04-17)

- All 8 clients across the network reported `finalized=3797, justified=3802` and stopped advancing. (Initial cause of the stall is not the subject of this issue — see caveat below.)
- Non-zeam clients continued to produce blocks; network head advanced to ~slot 12085.
- zeam's head was frozen at slot 3806. Its logs filled with:
```
[warning] Cache full (1024 blocks), rejecting future block 0x.. at slot 12078
[warning] Cache full (1024 blocks), rejecting block 0x.. at slot 12080
[error] error.UnknownSourceBlock producing aggregation for slot=3806
```
- curl `/lean/v0/checkpoints/justified` on zeam returned `slot=3800` while peers returned `slot=3802`.
- Recovery required \`spin-node.sh --restart-client zeam_0 --checkpoint-sync-url https://leanpoint.leanroadmap.org/lean/v0/states/finalized\`.

## Root cause analysis

Cache admission in \`pkgs/node/src/node.zig\`:

\`\`\`zig
// If cache is full, reject - proactive pruning on finalization keeps the cache bounded
if (self.network.fetched_blocks.count() >= constants.MAX_CACHED_BLOCKS) {
self.logger.warn(\"Cache full ({d} blocks), rejecting block ...\", ...);
return CacheBlockError.CachingFailed;
}
\`\`\`
(also mirrored in \`cacheFutureBlock\` at ~line 593)

Pruning is wired via \`prune_cached_blocks_fn\` and invoked only when finalization advances (\`chain.zig:1102\`). \`constants.zig\` defines:

\`\`\`zig
pub const MAX_CACHED_BLOCKS = 1024;
\`\`\`

The failure mode is a feedback loop:

1. Something causes finalization to stop advancing (consensus issue, network partition, proposer outage — outside this issue).
2. The cache stops being pruned and grows by ~1 entry per gossiped / future block.
3. After ~1024 slots without finalization, the cache is full.
4. From that point on every new block is rejected, including the ones the node would fetch to resolve the stall (sync requests, parent backfills).
5. The node becomes permanently out of sync; even if the rest of the network resumes finalizing, this node cannot catch up because it cannot ingest the blocks it needs.

The code comment *\"proactive pruning on finalization keeps the cache bounded\"* is only accurate in the happy path. It is precisely the pathological case (no finalization) where unbounded growth would happen, and the current mitigation (hard reject) converts a transient liveness issue into a permanent one.

## Suggested directions (non-prescriptive)

Any of these would break the feedback loop:

1. **Evict by slot, not reject.** When \`fetched_blocks.count() >= MAX_CACHED_BLOCKS\`, drop the oldest (or highest-slot orphan with no descendants) instead of refusing ingress.
2. **Time-based pruning independent of finalization.** Periodically evict entries older than N slots regardless of whether finality has advanced.
3. **Distinguish sync-response blocks from gossip.** Blocks fetched in response to our own \`blocks_by_root\` requests should not be refused — they are the recovery path. Consider a separate admission policy (or a separate cache) for in-flight sync responses.
4. **Trigger checkpoint-sync fallback automatically.** When the node has been \`behind_peers\` for more than X slots and the cache is full, attempt a forced re-sync from a configured endpoint instead of waiting for human intervention.

Option 3 alone would likely have been sufficient to recover this incident without any operator action.

## Reproduction

Any scenario producing a finality gap of >\`MAX_CACHED_BLOCKS\` slots while the node remains connected to peers that keep gossiping new blocks. Easiest synthetic repro: lower \`MAX_CACHED_BLOCKS\` to 64 in a devnet, take down >1/3 of validators for ~5 minutes, observe zeam nodes fail to recover even after the validators return.

## References

- \`pkgs/node/src/node.zig\` L537-545 (cacheBlock admission)
- \`pkgs/node/src/node.zig\` L593-600 (cacheFutureBlock admission)
- \`pkgs/node/src/constants.zig\` L16-18 (\`MAX_CACHED_BLOCKS = 1024\`)
- \`pkgs/node/src/chain.zig\` L1102-1106 (pruning invocation, finalization-gated)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with cacheBlock and cacheFutureBlock in pkgs/node/src/node.zig, then trace pruning from chain.zig and the MAX_CACHED_BLOCKS definition in constants.zig. Reproduce with a reduced cache limit and a prolonged finality gap. Done means the node can ingest the blocks needed to recover after finalization resumes, without the cache remaining permanently full.

Written by the indexing model from the issue text.

Assessment

Tech stack
zig
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.