blockblaz / blockblaz/zeam

node: parallel XMSS verify with serial fork-choice apply for gossip aggregations

Open
#895 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Zig
Stars
97
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Context

Fresh devnet restarts can flood the node with gossip aggregated attestations while `blocks_by_range` catch-up keeps the chain-worker busy on block STF. The aggregated-attestation queue (512) then fills faster than a single worker drains it, producing `aggregated attestation queue full, dropping slot=…` and starving fork-choice vote tracking.

PR #894 mitigates this with:
- buffer-on-`QueueFull` → `pending_aggregated_attestations` (leanSpec-style)
- batch drain (up to 32 aggs per loop when depth > 16)
- less per-agg work (single participant-bit decode)

Those are backpressure / throughput tweaks on the **existing** architecture: one chain-worker thread applies blocks, raw attestations, and aggregations in a fixed loop (block first, then one att, then one-or-many aggs).

## Problem

One applier thread is correct for **consistency** (fork choice, `states` map, historical roots must stay ordered), but it becomes the bottleneck when verify + apply are colocated on that thread under gossip burst + block catch-up.

We already parallelize some work elsewhere:
- local aggregation production: dedicated `aggregate_io` (#873)
- pending-attestation **replay**: thread pool for XMSS verify when configured
- libp2p / network: separate runtime

Incoming gossip aggregations still run **validate + XMSS verify + fork-choice store** entirely on the chain-worker thread.

## Proposed direction (not implemented here)

**Parallel verify, serial apply** — same logical model as leanSpec’s single store writer, but split CPU-bound verify from state mutation:

1. **Verify stage (parallel):** libxev or a small pool runs `validateAttestationData` + `verifyAggregatedAttestation` (XMSS) on cloned gossip payloads; output is a bounded queue of verified messages or an error (drop / buffer unknown-block for replay).
2. **Apply stage (serial):** the chain-worker (or a dedicated single “applier” thread) only runs `forkChoice.onAttestation` + `storeAggregatedPayload` for verified messages, in an order consistent with block imports (e.g. block queue still gates: no apply for targets whose state was pruned mid-import).

**Do not** split into three independent workers (blocks / raw att / agg) all mutating fork choice concurrently — that races STF, fork-choice head, and `historical_block_hashes` without a strict cross-queue ordering protocol.

### Alternative considered

Two chain-worker queues (blocks vs attestations) with one writer lock on fork choice — still serial apply, only helps scheduling; does not remove XMSS cost from the hot path unless verify is moved off-thread.

## Acceptance criteria

- [ ] Under devnet restart load, `lean_chain_queue_dropped_total{queue="aggregated_attestation"}` stays near zero without relying only on a larger queue.
- [ ] `lean_pending_attestations_buffered_total{reason="queue_full"}` remains bounded under sustained gossip.
- [ ] No regression in fork-choice / STF ordering (blocks still applied before dependent agg applies; replay buffer semantics unchanged).
- [ ] Document lock ordering if verify pool and chain-worker both touch `states` / fork choice (see `docs/threading_refactor_slice_a.md`, #803).

## Related

- #803 — BeamNode threading model refactor
- #863 — libxev / chain-worker move-off
- #890 — chain-worker queues and pending replay
- #893 / PR #894 — blocks_by_range catch-up + agg queue backpressure mitigations

## Notes

Raw gossip attestations could use the same verify-then-apply split later; start with aggregations (heavier XMSS, primary queue-full source on devnet).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading docs/threading_refactor_slice_a.md and the chain-worker queue and pending replay work referenced in #803, #890, and #893. Trace where aggregated gossip validation, XMSS verification, fork-choice application, and payload storage currently run. Done means parallel verification with bounded buffering, serial ordered application, preserved replay semantics, and documented lock ordering, validated under devnet restart load.

Written by the indexing model from the issue text.

Assessment

Tech stack
zig
Domain
distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.