blockblaz / blockblaz/zeam

node: aggregator-only freeze of slot-tick / onInterval loop after MissingPreState cascade (separate from #837 STF reject)

Open
#847 0 comments 0 reactions 1 assignee Claimed by @ch4r10t33r View on GitHub
bug
Dominant language
Zig
Stars
97
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Summary

When the zeam node is running with `--is-aggregator …`, a chain of `error.MissingPreState` retries on a single rejected ancestor block converts into a **permanent freeze of the time-advance loop**. A regular zeam validator on the same image, same network, hitting the **same** rejected block and the **same** `MissingPreState` retry pattern, recovers cleanly within the same slot window.

The freeze is at the slot-tick / `onInterval` layer, not at the network or fork-choice layer: the wedged aggregator continues to receive gossip blocks, dispatch RPCs, and update its internal `Latest Justified` view; only the slot counter and the interval cycle stop advancing.

## Why it deserves a separate issue from #837

#837 covers the upstream trigger — a transient `error.DuplicateAttestationData` STF reject on a block accepted by every other client family. That is a question about **attestation-dedup correctness in the STF**.

This issue is about a **separate downstream defect**: the zeam aggregator's slot-tick loop is not resilient to a sequence of `MissingPreState` errors on the chain-worker. Even if the STF false-positive in #837 were fixed tomorrow, **any** real `MissingPreState` cascade (e.g. legitimate orphan fork, slow RPC import, pruning-window edge) would re-produce the same wedge, because the time-advance loop conflates STF/aggregation errors with "do not advance time".

This is the same broader-resilience bug-class flagged in #749 / #776: STF errors on a block must reject the offending block, not freeze the slot/interval clock.

## Reproduction (devnet-4, 2026-05-07 20:11–21:30 UTC)

Fresh-genesis devnet boot at 20:11:14 UTC. 16 nodes (5 client implementations + zeam ×2). All on `blockblaz/zeam:devnet4` digest `sha256:678dd5b8…62b51` (post-#846, the v0.4.17 build).

A single block at slot 100, proposer=4 (qlean_0), `parent_root=0xdf7c…0899f`, was rejected by zeam STF as `error.DuplicateAttestationData`. **0 of 14 non-zeam containers** rejected this block. The two zeam nodes rejected it 23 ms apart.

After the reject, the slot 101 cascade was identical on the two zeam nodes:

```
zeam_0 [s=101 i=0] error.MissingPreState (20:13:36.049)
zeam_0 [s=107 i=2] error.MissingPreState (20:14:01.638)
zeam_0 [s=117 i=2] error.MissingPreState (20:15:15.183)
zeam_0 [s=118 i=2] error.UnknownHeadBlock (20:15:18.669) ← from aggregation path
zeam_0 [s=139 i=2] error.UnknownSourceBlock(20:16:20.121) ← from aggregation path
zeam_0 [s=139 i=2] … frozen for 70+ minutes … ← time loop wedged

zeam_1 [s=101 i=0] error.MissingPreState (20:13:36.019)
zeam_1 [s=107 i=2] error.MissingPreState (20:14:01.612)
zeam_1 [s=113 i=2] error.MissingPreState (20:14:25.897)
zeam_1 [s=119 i=2] error.MissingPreState (20:14:50.344)
zeam_1 [s=125 i=3] error.MissingPreState (20:15:14.682)
zeam_1 [s=131 i=4] error.MissingPreState (20:15:39.306)
zeam_1 [s=138 i=0] error.MissingPreState (20:16:04.347)
zeam_1 … recovered, justified=292 by 21:30 UTC …
```

Same image, same input, only the aggregator role differs.

## Aggregator vs regular validator side-by-side

| | zeam_0 | zeam_1 |
|---|---|---|
| Image | `blockblaz/zeam:devnet4` (post-#846) | `blockblaz/zeam:devnet4` (post-#846) |
| `--is-aggregator` | yes | no |
| `--aggregate-subnet-ids` | `0,1` | (none) |
| Slot 100 STF reject | yes (`DuplicateAttestationData`) | yes (`DuplicateAttestationData`) |
| `MissingPreState` retries on slot 101 | 4 | 8 |
| `error.UnknownSourceBlock` from aggregation path | **154x** | 0 |
| `error.UnknownHeadBlock` from aggregation path | 5x | 0 |
| `error.MissingState` from sig-aggregation path | 11x | 0 |
| `[clock] slot_interval=4 duration=…` lines after boot | **0** (never reached interval 4 again) | many (clock cycles normally) |
| Final state at 21:30 UTC | wedged at `[s=139 i=2]` since 20:16:20 | healthy, justified=292, fully caught up |
| Local API `lean/v0/checkpoints/justified` | `slot=121` (frozen) | `slot=292` (live) |

## Why the aggregator is special

Three error classes are aggregator-only on zeam and they all fire from the per-interval aggregation production path. Only the aggregator runs them; the regular validator does not:

```
[chain] failed to aggregate attestation signatures for slot=N: error.MissingState ← sig-aggregation
[node] error producing/publishing aggregations at slot=N interval=K: error.UnknownSourceBlock
[chain] invalid aggregated attestation data in block: error=error.UnknownHeadBlock
```

These run every interval 2 of every slot. After the STF cascade leaves an ancestor block missing, the aggregator path tries to look up that ancestor, fails, and (this is the bug) **the failure breaks something in the slot-tick loop** rather than just dropping the aggregation for that slot.

Empirically, the zeam_0 clock log shows the freeze:

```
$ {container="zeam_0"} |~ '\[clock\] slot_interval=4 duration' → 0 hits
$ {container="zeam_0"} |~ '\[clock\] slot_interval=2 duration' → many (looping at slot 139)
```

The interval cycle never advances past 2 within slot 139 — the loop fires intervals 0, 1, 2, then loops back to interval 0 of the **same** slot 139 instead of advancing to interval 3 / interval 4 / slot 140. After 70 minutes of this, the slot counter has gone nowhere.

Meanwhile the network plane is fine — gossip is still received, RPC requests still flow, the `Latest Justified` view in chain-status even continues to advance from 110 to 121 because incoming gossip + fork-choice keeps updating it. The time loop is the only thing that's frozen.

## Asks

1. **Decouple the slot-tick advance from STF and aggregation errors.** An error during `aggregateAttestationSignatures`, `produceAggregation`, or `publishAggregation` for slot N must not block the next interval's tick. Drop the aggregation, log it, advance the clock.

2. **Drain `MissingPreState` retries with a bounded budget.** A few retries are fine; an unbounded retry of the same descendant slot every interval (zeam_0 retried slot 101 four times in 100 seconds and then gave up by way of wedging) should either give up explicitly with a `[chain] giving up on slot 101 root=0x…` log, or evict the orphan from the worker queue.

3. **Add a `lean_clock_loop_stalled_seconds` (or similar) gauge** that tracks the wall-clock delta between "expected slot from genesis time" and "actual slot we last ticked". Right now the only way to detect this freeze externally is to scrape the API and notice the slot number isn't moving — there's no internal signal.

## Cross-references

- #837 — STF false-positive on slot 100 (separate question: is `DuplicateAttestationData` correctly raised? This issue assumes the cascade can happen and asks for resilience regardless).
- #788 — fork-choice reset under timing desync. **Fixed by #841** (buffer future-slot gossip during clock lag). Distinct from this bug — #788 was about clock desync rejecting future-slot blocks; this is about a chain-worker error cascade freezing the clock itself.
- #749 / #776 — broader theme: STF errors must reject blocks, not freeze fork-choice / time.
- Original observation in the now-superset comment on [#837 (recurrence on 2026-05-07)](https://github.com/blockblaz/zeam/issues/837#issuecomment-4400838389) — splitting the freeze portion into this issue per maintainer request.

## Logs

- Loki: `http://46.225.10.32:3100/`
- zeam_0: `{container="zeam_0"}` 20:11:14 UTC → present (still wedged)
- zeam_1: `{container="zeam_1"}` 20:11:14 UTC → present (recovered, healthy)
- Diagnostic queries:
```
{container="zeam_0"} |~ "\\[clock\\] slot_interval=4" → 0 hits
{container="zeam_0"} |~ "error producing/publishing aggregations" → 154 hits
{container="zeam_0"} |~ "error.MissingPreState" → 3 hits (no further attempts after wedge)
{container="zeam_1"} |~ "error producing/publishing aggregations" → 0 hits
{container="zeam_1"} |~ "error.MissingPreState" → 8 hits then 0 (recovered)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.