BeamNode threading model refactor — 8-point plan (supersedes #798-#802)

## Context

Refactor the BeamNode threading model to address the 8-point plan agreed in the 2026-04-29 mutex/parallelism review (Gajinder, Anshal, Partha):

> 1. All resources shared across threads maintain their own locks/mutexes (forkchoice is a resource).
> 2. Minimize locking — e.g. serving req/resp shouldn't need a lock except reading a forkchoice snapshot.
> 3. Only finalization advancement might need a lock across multiple resources because we do a canonicality view and then prune multiple resources.
> 4. All onBlock operations to be maximally parallel — e.g. sig verification can be parallel with pre-state getting cloned/ready.
> 5. Compute hash roots once for gossip objects and cache them there; reuse from there.
> 6. onBlock followups on a separate thread so onGossip can be freed up faster (followup is not critical).
> 7. Remove onBlockFollowup from any backfilling operations.
> 8. Network fetching and pruning of missed roots should be a parallel op.

This issue supersedes the BeamNode-mutex hoist stack #798 → #799 → #800 → #801 → #802. Those PRs were good devnet fire-fighting (good metrics, real wins) but they all work *around* the existing single `BeamNode.mutex`, while points 1+2 ask for a different shape: per-resource locks + lock-free read paths.

We are closing 798–802 in favor of slicing this refactor vertically, so each slice is independently reviewable and consensus-correct on its own.

## Devnet metrics carried forward from the closed stack

These came out of the #786 mutex instrumentation and motivate the work; preserved here so the diagnoses aren't lost.

### From #798 (gossip block import — `chain.onBlock` verify+STF)

| site | hold sum | hold mean | hold p99 (events >2s) |
|------|----------|-----------|------------------------|
| **onGossip** | **496.83s** | **49ms** | **122 events >2s** |
| onReqRespResponse | 71.85s | 23ms | 13 events >2s |
| onReqRespRequest.blocks_by_root | 8.98s | 20ms | 0 |
| onInterval | 4.83s | 2.6ms | 0 |

`onInterval` wait sum 13.75s (mean 7.4ms, p99 >100ms) — interval ticks delayed by gossip block imports holding the mutex. This drives the `slot_interval=1 duration=1.071s` pattern.

`zeam_chain_onblock_duration_seconds` mean 77ms (32s sum / 413 blocks).

### From #799 (gossip attestation verify)

| metric | sum | count | mean |
|--------|-----|-------|------|
| `lean_attestation_validation_time_seconds` | **383s** | 11,355 | **33ms** |
| `lean_pq_sig_aggregated_signatures_verification_time_seconds` | 25s | 426 | 59ms |

~408s of mutex hold attributable to gossip attestation verify alone over a ~30 minute session.

### From #800 (block production aggregation+STF)

| metric | sum | count | mean |
|--------|-----|-------|------|
| `lean_block_building_time_seconds` | 12.4s | 15 | **826ms** |
| `lean_block_building_payload_aggregation_time_seconds` | 12.07s | 15 | 805ms |

97% of block-building time is the aggregation step. Today it runs under `BeamNode.mutex` on the i=0 onInterval thread — single-handedly stalls 1+ libxev tick per proposal slot.

### From #801 (backfill replay)

`processPendingBlocks` was the last call site explicitly passing `null` for the external mutex despite running under it. Each replayed block carried the full ~77ms onBlock cost under the lock.

## Vertical slice plan

Each slice is a standalone PR, independently reviewable, with its own devnet verification before the next slice starts.

### Slice (a) — Per-resource locks + drop `BeamNode.mutex` from req/resp **[STARTING NOW]**

Scope:
- Audit shared resources accessed across threads. Initial list:
- `forkChoice` — already has `RwLock` ✅
- `states` map — currently protected by `BeamNode.mutex` only
- `pending_blocks` — currently protected by `BeamNode.mutex` only
- `public_key_cache`, `root_to_slot_cache` — documented not thread-safe, mutex-protected
- `last_emitted_*` checkpoints — currently single-writer (only chain itself writes)
- Add explicit per-resource locks where the resource is shared.
- Drop `BeamNode.mutex` from `onReqRespRequest` / `onReqRespResponse`. Req/resp serves block lookups — these only need a forkchoice snapshot read (which the RwLock already supports lock-free for readers in practice) and a DB lookup (DB is already serialized by its own backend).
- Keep `BeamNode.mutex` (or rename it) for the narrow case G called out: finalization advancement, where we hold a multi-resource lock during canonical-view-then-prune.

Risk: highest of the 5 slices — req/resp is hot, getting the forkchoice snapshot read wrong = serving wrong-fork blocks. Will mark draft, request explicit review.

### Slice (b) — onBlock pipeline parallelism

Scope:
- Restructure `chain.onBlock` into staged pipeline:
- Stage 1 (locked, fast): parse, validate, dedup against forkchoice
- Stage 2 (parallel): hash root + sig verify ‖ pre-state clone — both are pure CPU on independent inputs; today they run sequentially under the lock-dance
- Stage 3 (locked, fast): STF runs on cloned state, no shared state mutation
- Stage 4 (locked, fast): forkchoice.onBlock + states.put + db write
- Stage 5 (dispatched): followup
- Use the existing `ThreadPool` / `spawnWg` infrastructure already wired into `chain.thread_pool`.

Risk: medium — STF is deterministic on cloned state, parallelism boundary is clean.

### Slice (c) — Followup worker thread

Scope:
- Single-consumer worker thread + bounded MPSC queue on `BeamNode`.
- Wires through `chain.setFollowupDispatch` (the seam from #802, which we re-introduce as part of this slice).
- Phase B finalization-followup runs off-thread; gossip returns as soon as fc.onBlock + states.put commit.
- Backfill (`processPendingBlocks`) runs a single trailing followup-dispatch instead of per-block, also from #802.

Risk: medium — careful about: (1) thread-safety of forkchoice prune writes, (2) `db.commit` from worker thread, (3) `event_broadcaster` fan-out from worker, (4) `prune_cached_blocks_fn` callback which reaches into BeamNode.

### Slice (d) — Parallel net-fetch + missed-root prune

Scope:
- Today `fetchBlockByRoots` and the in-flight pruning / dedup of missing roots are sequential. Under high block-arrival rate, this serializes a CPU-bound dedup behind a network IO call.
- Run them concurrently: spawn the rpc dispatch on the network thread, do the dedup/prune of already-known roots in parallel.

Risk: low — both ops are localized to `BeamNode.fetchBlockByRoots` and the `network.ensureBlocksByRootRequest` path.

### Slice (e) — Centralised gossip envelope hash-root cache

Scope:
- `networks.GossipMessage` (block / attestation / aggregation) computes `hashTreeRoot` once at gossip ingress and caches the result on the envelope.
- Every downstream consumer (`chain.onGossip`, `forkChoice`, db key, dedup checks) reads the cached root instead of recomputing.
- Today the block path precomputes `block_root` in `BeamNode.onGossip` before taking the lock (post-#786) but it's not centralised — req/resp recomputes, attestation paths recompute, etc.

Risk: low-medium — touches the `GossipMessage` shape; needs to be coordinated with the rust libp2p bridge (#789) so we don't break the FFI.

## Closing the prior stack

#798 #799 #800 #802 are being closed with a pointer to this issue. #801 was already stacked on the rest. The diagnoses + devnet metrics from each PR description are preserved above.

The instrumentation PR #786 / #787 stays — that's how we measure progress on each slice.

cc @gr3999 @anshalshukla @ch4r10t33r

Begin with Slice (a): audit forkChoice, states, pending_blocks, public_key_cache, and root_to_slot_cache, then read onReqRespRequest/onReqRespResponse and the existing forkChoice RwLock. Verify the per-resource locking boundaries before changing the req/resp paths. Done means req/resp no longer relies on BeamNode.mutex while finalization retains the narrow multi-resource lock, with devnet verification and explicit review.

Written by the indexing model from the issue text.