bug: consensus stall

Bug 2: Finalization Halts When One Node Is Down

Symptom

After node3 becomes a zombie (Bug 1), the remaining 5 healthy nodes continue M-notarizing (progressing views at ~50ms/view), but the finalized block height stops advancing (stuck at height 8085). The
gRPC get_latest_block endpoint returns the same block indefinitely because it reads from the FINALIZED_BLOCKS storage table, which is only written to on L-notarization.

This is reproducible — restarting all nodes produces the same pattern: rapid finalization initially, then it halts permanently once any node falls behind.

Background: Minimmit Consensus Thresholds (N=6, F=1)
┌────────────────┬───────────┬──────────────┬──────────────────────────────────────┐
│ Event │ Threshold │ Votes Needed │ What It Does │
├────────────────┼───────────┼──────────────┼──────────────────────────────────────┤
│ M-notarization │ >2F │ 3 │ Advances to next view (liveness) │
├────────────────┼───────────┼──────────────┼──────────────────────────────────────┤
│ L-notarization │ ≥N-F │ 5 │ Finalizes block, persists to storage │
└────────────────┴───────────┴──────────────┴──────────────────────────────────────┘
With one node down, there are exactly 5 voters — the bare minimum for L-notarization.

The Race Condition

The protocol advances views on M-notarization, not L-notarization. This creates a fundamental race:

1. View V starts. Leader proposes a block. 5 healthy nodes begin voting.
2. 3rd vote arrives at a node → M-notarization threshold crossed → view immediately advances to V+1. The node starts processing V+1 messages.
3. 4th and 5th votes for view V arrive after view advancement. These are "late votes" for a now-old view.

The code IS designed to handle late votes: old views are kept in a non_finalized_views map, and late votes are routed there via view_chain.route_vote(). When the 5th vote arrives, handle_vote should
return ShouldFinalize, triggering finalize_with_l_notarization() which persists the block.

Why Late-Vote Finalization Fails In Practice

There are several interacting failure modes that prevent the late-vote path from working reliably:

1. Vote rejection for non-M-notarized views (view_chain.rs:395-397)

When routing a vote to a non-current view, the code calls:
if view_number != self.current_view {
ctx.has_view_progressed_without_m_notarization()?;
}
If this check fails (e.g., the view was nullified, or the M-notarization hasn't been stored in this node's context yet due to message ordering), the vote is rejected with an error. The logs confirm
this:
ERRO Error handling consensus message: M-notarization not found for view 5048
This means legitimate late votes for old views are being dropped, reducing the effective vote count below the L-notarization threshold.

2. View nullification breaks the chain

Every 6th view has node3 as leader (round-robin). Node3 can't propose → 5-second timeout → nullification. Nullified views have no M-notarization and no block, so they can never be L-notarized. More
importantly, they break the contiguous finalization chain: finalize_with_l_notarization persists all views from the oldest non-finalized up to the target view. If any intermediate view is nullified and
can't be finalized, it may block the chain.

3. Cascading failure

Once finalization falls behind, the non_finalized_views map grows (one entry per view). Views keep advancing on M-notarization (only needs 3 votes, easy to achieve). But finalization never catches up
because of (1) and (2). The gap between "current view" and "last finalized view" grows without bound.