blockblaz / blockblaz/zeam

[bug] Locally produced aggregations fail with `error.UnknownSourceBlock`, causing node to stall

Open
#752 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Zig
Stars
97
Forks
39
PR merge metrics
No merged PRs in 30d

Description

During the devnet-4 run (Apr 16, 2026), the following error was observed on `zeam_0` at slot 3806:

```
[error] (zeam): [node] error producing/publishing aggregations at slot=3806 interval=19032: error.UnknownSourceBlock
```

The node then entered an infinite loop re-publishing stale aggregations for slots 3805/3806, never advancing past slot 3806, while the rest of the network progressed normally.

---

**Root cause**

`publishAggregation` passes locally produced aggregations through `chain.onGossipAggregatedAttestation`, which runs the same validation as incoming gossip — including a check that `attestation.data.source.root` exists in the forkchoice proto array:

```zig
// chain.zig
const source_block = self.forkChoice.getProtoNode(data.source.root) orelse {
return AttestationValidationError.UnknownSourceBlock;
};
```

The attestation is constructed using `forkChoice.getLatestJustified()` as the source checkpoint. Between construction time and validation time, the forkchoice can be rebased (pruned), causing the source root to disappear from the proto array. Since the aggregation was built from local state, this validation failure is a false negative — the attestation was valid when produced.

Additionally, when `publishProducedAggregations` returns this error, `onInterval` returns early **without updating `self.last_interval`**:

```zig
self.publishProducedAggregations(aggregations) catch |e| {
self.logger.err("error producing/publishing aggregations at slot={d} interval={d}: {any}", ...);
return e; // last_interval NOT updated
};
```

On the next tick, `start_interval = self.last_interval + 1` replays the same failed interval, producing the same stale aggregations, hitting the same error — indefinitely.

---

**Impact**

- Node stalls at the slot where the first failure occurs
- Validators managed by that node stop attesting **and stop aggregating**
- If the stalled node was the sole aggregator, finalization halts even if enough individual validators are still online

In this run, `zeam_0` held **1 of 8 validators** and was acting as the **sole aggregator** for the network — collecting individual attestations from all active peers each slot and publishing combined aggregates. When it stalled at slot 3806, no other node took over aggregation. Block producers stopped receiving aggregated attestations sufficient to cross the 2/3 finalization threshold, causing finality to freeze at slot 3797.

---

**Proposed fix**

Two independent fixes are needed:

1. **Skip gossip validation for locally produced aggregations.** `publishAggregation` should not call `onGossipAggregatedAttestation`. Locally produced aggregations are already validated at construction time and should be added directly to the chain without re-running gossip checks.

2. **Handle `UnknownSourceBlock` gracefully in `onInterval`.** Even if validation is kept, a failure to publish one aggregation should not stall the entire interval loop. The error should be logged and skipped, and `last_interval` must be updated regardless.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in chain.zig with publishAggregation, onGossipAggregatedAttestation, and onInterval, then trace publishProducedAggregations and the last_interval update. Reproduce or inspect the UnknownSourceBlock path and confirm that locally produced aggregations do not trigger the false failure and that one publishing error cannot replay an interval indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
zig
Domain
blockchain, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.