[bug] Locally produced aggregations fail with `error.UnknownSourceBlock`, causing node to stall
- Dominant language
- Zig
- Stars
- 97
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
During the devnet-4 run (Apr 16, 2026), the following error was observed on `zeam_0` at slot 3806:
```
[error] (zeam): [node] error producing/publishing aggregations at slot=3806 interval=19032: error.UnknownSourceBlock
```
The node then entered an infinite loop re-publishing stale aggregations for slots 3805/3806, never advancing past slot 3806, while the rest of the network progressed normally.
---
**Root cause**
`publishAggregation` passes locally produced aggregations through `chain.onGossipAggregatedAttestation`, which runs the same validation as incoming gossip — including a check that `attestation.data.source.root` exists in the forkchoice proto array:
```zig
// chain.zig
const source_block = self.forkChoice.getProtoNode(data.source.root) orelse {
return AttestationValidationError.UnknownSourceBlock;
};
```
The attestation is constructed using `forkChoice.getLatestJustified()` as the source checkpoint. Between construction time and validation time, the forkchoice can be rebased (pruned), causing the source root to disappear from the proto array. Since the aggregation was built from local state, this validation failure is a false negative — the attestation was valid when produced.
Additionally, when `publishProducedAggregations` returns this error, `onInterval` returns early **without updating `self.last_interval`**:
```zig
self.publishProducedAggregations(aggregations) catch |e| {
self.logger.err("error producing/publishing aggregations at slot={d} interval={d}: {any}", ...);
return e; // last_interval NOT updated
};
```
On the next tick, `start_interval = self.last_interval + 1` replays the same failed interval, producing the same stale aggregations, hitting the same error — indefinitely.
---
**Impact**
- Node stalls at the slot where the first failure occurs
- Validators managed by that node stop attesting **and stop aggregating**
- If the stalled node was the sole aggregator, finalization halts even if enough individual validators are still online
In this run, `zeam_0` held **1 of 8 validators** and was acting as the **sole aggregator** for the network — collecting individual attestations from all active peers each slot and publishing combined aggregates. When it stalled at slot 3806, no other node took over aggregation. Block producers stopped receiving aggregated attestations sufficient to cross the 2/3 finalization threshold, causing finality to freeze at slot 3797.
---
**Proposed fix**
Two independent fixes are needed:
1. **Skip gossip validation for locally produced aggregations.** `publishAggregation` should not call `onGossipAggregatedAttestation`. Locally produced aggregations are already validated at construction time and should be added directly to the chain without re-running gossip checks.
2. **Handle `UnknownSourceBlock` gracefully in `onInterval`.** Even if validation is kept, a failure to publish one aggregation should not stall the entire interval loop. The error should be logged and skipped, and `last_interval` must be updated regardless.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in chain.zig with publishAggregation, onGossipAggregatedAttestation, and onInterval, then trace publishProducedAggregations and the last_interval update. Reproduce or inspect the UnknownSourceBlock path and confirm that locally produced aggregations do not trigger the false failure and that one publishing error cannot replay an interval indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- zig
- Domain
- blockchain, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100