node, aggregation: zeam aggregate-build is 5–7x slower than ethlambda on devnet (2–3s vs 0.4s)

## Summary

On the multi-client devnet (8 aggregators across `zeam` / `ethlambda` / `ream`), zeam's per-aggregate **build** time is **5–7× higher** than ethlambda's, measured cross-client at the same wall-clock with the same standard `lean_pq_sig_aggregated_signatures_building_time_seconds` histogram.

| Aggregator | mean build time | sample count | head_slot |
|---|---:|---:|---:|
| zeam_8 | **2.90 s** | 64 | 256 |
| zeam_4 | **2.91 s** | 48 | 227 |
| zeam_16 | **2.00 s** | 64 | 261 |
| ethlambda_8 | 0.43 s | 88 | 271 |
| ethlambda_5 | 0.41 s | 8 | 108 |
| ethlambda_16 | 0.40 s | 12 | 263 |
| ream_0 | _(metric not exposed — see Notes)_ | — | 223 |
| ream_13 | _(metric not exposed)_ | — | 215 |

(All numbers from each host's local Prometheus, snapshot `2026-05-19T15:13Z`, ~30 min after fresh devnet restart.)

Verify side is fine across all clients (`aggregated_signatures_verification_time` is 35–71 ms; `attestation_verification_time` is 0.78–1.25 ms). The cost gap is squarely on the **build** path.

`drops` and queue saturation are clean on this run (`lean_chain_queue_dropped_total = 0` for `block` and `attestation` queues on every zeam aggregator), so the slowness is **not** a queue-backpressure artifact and is **not** the regression PR #894 fixed. It's intrinsic per-aggregate cost.

## Why it matters — direct FFG impact

Slow build → zeam aggregators frequently miss a slot's aggregation window (`zeam_aggregate_skip_total{reason="in_flight"}` ticks up because `aggregate_group.concurrent` is gated at `concurrent_limit=1`, see `pkgs/node/src/chain.zig:4357`). The downstream effect is visible directly in the next block proposal's coverage. From `zeam_8` log at slot 256:

```text
[forkchoice] block proposal slot=256: attestation aggregate coverage:
payloads = subnet0=3/8(37.50%) subnet1=8/8(100%) subnet2=7/8(87.50%)
subnet3=8/8(100%) subnet4=7/8(87.50%) subnet5=1/8(12.50%)
subnet6=6/8(75%) subnet7=6/8(75%)
network=46/64(71.88%)
```

Subnets 0, 3, 5 are the three **zeam-aggregator** subnets. Subnet 5 contributed **1 of 8** votes. Subnet 0 contributed **3 of 8**. Network coverage hovers around 70–72%, marginal vs the 67% justification threshold once any vote is on a divergent target — currently `latest_justified_slot=0` and `latest_finalized_slot=0` for **every** node in the devnet.

So this issue is the dominant single contributor to "no FFG progress" on the current cross-client devnet (other contributors — #863 slot-driver backlog, head divergence — are already tracked).

## Where the cost lives

- Entry: `chain.zig` `submitAggregateOnInterval` → `aggregate_group.concurrent` on the dedicated `aggregate_io` thread (the move done in #873).
- Worker: `chain.zig` `aggregateImpl` (`pkgs/node/src/chain.zig:4371`) →
`forkChoice.aggregate` → `aggregateUnlocked` (`pkgs/node/src/forkchoice.zig:2035`).

`aggregateUnlocked` is structured in three phases:

1. **Snapshot** under `signatures_mutex` — `AggregateSnapshot.takeUnderLock`. Comment claims `O(10 ms)` typical.
2. **Compute** with no lock held — `agg.computeAggregatedSignatures(&state.validators, &snap.signatures, &snap.new_payloads, &snap.known_payloads)`. This is the heavy XMSS FFI path. The same docstring on `aggregate()` at line 2240 says "*The ~18 s `computeAggregatedSignatures` runs in between with no locks*" — i.e. the worst case acknowledged in code is **~18 s**, and the live mean we measure today is **2–3 s**.
3. **Commit** under `signatures_mutex` — merge results, prune consumed snapshot vids. ms-scale.

So the work that's slow is **phase 2 only** — the FFI into `multisig-glue` for XMSS aggregation/proof-construction. Phases 1 and 3 are not the cost drivers.

ethlambda doing the equivalent work in 0.4 s is the strongest signal that this is **not** a fundamental cost of the spec — there is something specific to zeam's multisig FFI path that is 5–7× off.

## Notes on cross-client comparability

- `lean_pq_sig_aggregated_signatures_building_time_seconds` is part of the standard leanMetrics set, so the histogram name and semantics match across zeam / ethlambda by construction.
- ream **does not expose** this histogram (or `_total`, `_invalid_total`, `_in_aggregated_signatures_total`, or the individual `attestation_verification_time` histogram). Their `/metrics` only carries `_valid_total` + `_verification_time_seconds` for aggregated sigs. That very likely means ream does not generate a recursive/SNARK aggregation proof at all (just packs verified XMSS sigs + a participation bitfield), and therefore has nothing to time. Worth confirming with the ream team before drawing conclusions, but it does suggest that even a SNARK-free packing is spec-acceptable.

## Suggested investigation

1. **Profile `multisig-glue` aggregation FFI** under a representative load (8 raw atts, validator set 64) on a single host. Identify whether the cost is in:
- per-signature recursive-proof witness construction (parallelisable per signature)
- shared setup / SRS load (cacheable, currently re-done?)
- bigint / hash-chain work (constant factor)
- any debug-mode build artefact in the rust crate that landed on the devnet image.
2. **Compare the multisig-glue API surface with what ethlambda uses.** If they use a different (lighter) aggregation construction, decide whether to align — or document the cost/security trade-off.
3. **Check parallelism.** `aggregate_io` is concurrent_limit=1 (#873), so two slots' aggregations cannot run in parallel, but within a single aggregation the per-signature work could be parallelised across a thread pool. Currently `computeAggregatedSignatures` appears to call FFI sequentially per attestation; verify and parallelise if so.
4. **Verify Rust release-mode / opt-level on the devnet build.** A 5–7× hit is consistent with debug-mode FFI, so rule that out first.
5. **Add per-phase timing** to `aggregateUnlocked` — break the histogram into `phase1_snapshot`, `phase2_compute_ffi`, `phase3_commit` so we can tell from Prometheus alone whether the cost is in the FFI or in surrounding Zig work.

## Acceptance criteria

- [ ] Mean `lean_pq_sig_aggregated_signatures_building_time_seconds` on a zeam aggregator ≤ 0.6 s on the multi-client devnet (matching ethlambda within ~1.5×).
- [ ] On a 32-validator-per-subnet stress run, p99 build time ≤ 2 s.
- [ ] `zeam_aggregate_skip_total{reason="in_flight"}` rate on a zeam aggregator drops by ≥ 4× compared to today's devnet baseline.
- [ ] Per-phase build histogram exposed (`phase1_snapshot`, `phase2_compute_ffi`, `phase3_commit`) so we can attribute future regressions.
- [ ] Block-proposal `aggregate coverage` from a zeam-managed subnet (0, 3, 5) reaches ≥ 6/8 in steady state on the same devnet.

## Related

- #873 — *(closed)* off-load aggregate from libxev thread. This issue is downstream of that — even on the dedicated `aggregate_io` thread, the per-aggregate cost itself is 5–7× ethlambda.
- #895 — parallel **verify** with serial fork-choice apply for incoming gossip aggregations. Symmetric concern on the verify path; this issue is the produce path.
- #863 — slow `slot_interval` / libxev event-loop starvation. Slow aggregate **build** is one of the largest contributors to the kind of long FFI completions #863 is investigating, even now that build has been moved off the libxev thread.
- #847 — aggregator-only freeze after MissingPreState cascade. Different defect, same surface (aggregator behaviour under load).

## Devnet repro context

- Branch under test on zeam aggregators: PR #894 (commit `b2654679`).
- Date: 2026-05-19, devnet restarted at ~14:43 UTC.
- Topology: 24 zeam + 24 ethlambda + 16 ream validators across 8 subnets (`gean` and `nlean` slots replaced with zeam/ethlambda in `lean-quickstart` `ansible-devnet/genesis/validator-config.yaml`).
- Live at snapshot time: head spread 108–271, jst=0, fin=0 on every aggregator.