zeam_0: rust-libp2p layer panics in delay_map (hashmap_delay.rs:94:46) — 21 crashes in 70 min, leaves node wedged in fc_initing

## Summary

Between 2026-05-07 10:14 and 11:25 UTC, `zeam_0` panicked **21 times** with the same signature in the `delay_map` crate — used by the rust-libp2p network layer. Each panic killed the zeam process; Docker's `--restart unless-stopped` policy respawned the container; each respawn re-ran `--checkpoint-sync-url` boot path. After the panics stopped, `zeam_0` was wedged in `fc_initing` (forkchoice still initing, awaiting first justified checkpoint) for ~100 minutes until it was externally `docker rm -f`'d at 13:07 UTC.

`zeam_1`, identical image (`blockblaz/zeam:devnet4`) on the same network in the same time window, **never panicked** (0 panics in Loki) and ran cleanly with head only 13 slots behind current at the time it was unrelatedly stopped.

## Panic signature

```
thread '' (1) panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/delay_map-0.4.1/src/hashmap_delay.rs:94:46:
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5, aborting
library/std/src/sys/pal/unix/mod.rs:305:14: 0x24d7a29 in abort_internal …
library/std/src/process.rs:2565:5: 0x24d4498 in abort …
library/std/src/rt.rs:57:13: 0x24c2b6a in rust_panic …
library/std/src/panicking.rs:850:5: 0x24d4814 in panic_with_hook …
library/std/src/panicking.rs:698:13:0x24c1f41 in {closure#0} …
library/std/src/sys/backtrace.rs:182:18: 0x24b7358 in __rust_end_short_backtrace …
library/std/src/panicking.rs:689:5: 0x24c2d9c in panic_handler …
library/core/src/panicking.rs:80:14: 0x24e6beb in panic_fmt …
```

Crate / line: [`delay_map-0.4.1` `src/hashmap_delay.rs:94:46`](https://docs.rs/delay_map/0.4.1/src/delay_map/hashmap_delay.rs.html#94) — the `update` call on the inner `DelayQueue` inside `HashMapDelay::insert`. This panics when the `delay_queue::Key` being updated has already been removed (queue/HashMap desync).

`fatal runtime error: failed to initiate panic, error 5, aborting` indicates the panic was raised while another panic was already unwinding (panic-during-panic), so the application frames between zeam's call site and the `delay_map` line could not be unwound — only the std-panic-machinery frames are present.

## Panic timeline (zeam_0, 2026-05-07)

```
10:10:08 --restart-client zeam_0 with --checkpoint-sync-url, container starts
10:14:53 panic #1 ← 4 min after checkpoint-sync restart
10:57:01 panic #2 ← gap explained by Docker's restart back-off
10:58:05 panic #3
10:59:41 #4
11:00:13 #5
11:01:17 #6
11:03:25 #7
11:05:01 #8
11:07:09 #9
11:08:45 #10
11:10:53 #11
11:13:01 #12
11:14:37 #13
11:15:41 #14
11:16:13 #15
11:16:45 #16
11:17:49 #17
11:18:53 #18
11:19:25 #19
11:21:01 #20
11:25:17 #21 ← last panic
… 11:25 → 13:07 … ← node wedged in fc_initing, no panics, no progress
13:07:36 external docker rm -f zeam_0 (Ansible)
```

After ~70 minutes of crash-loop, the node settled into a steady state where the panicking code path was no longer exercised because forkchoice never left `fc_initing`. From the chain status log just before removal:

```
CHAIN STATUS: Current Slot: 3150 | Head Slot: 573 | Behind: 2577
Latest Justified: Slot 573 | Root: 0xf7d61b20…
Latest Finalized: Slot 573 | Root: 0xf7d61b20…
ForkChoice Tree: f7d6(573)
```

The forkchoice tree had collapsed to a single block (the finalized anchor), and validator/aggregator action was skipped every interval:

```
[validator] skipping attestation production for slot=…: forkchoice still initing (awaiting first justified checkpoint)
[chain] skipping aggregation production for slot=…: forkchoice initializing
```

Loki count over the last 5 min before removal: **75 fc_initing skips** on zeam_0; **0** on every other node.

## Why this matters

`zeam_0` is the subnet-0 aggregator on devnet-4. While it's wedged:
- Subnet-0 attestations are not aggregated by zeam_0 (only ethlambda_1's `--aggregate-subnet-ids 0,1` covers it as fallback).
- Aggregation deadline pressure on the lone fallback rises — observed `aggregation_deadline_ms=750` cancellations on ethlambda_1 with `groups_aggregated=2 of 703 considered`.
- Justification cadence drops, the chain takes longer to traverse the next `is_justifiable_after` slot, and 3sf-mini finalisation gets harder to advance.

So a single misbehaving aggregator (zeam_0) has secondary chain-wide consequences (finality stall) even though the rest of the cluster is healthy.

## Reproduction notes

- Image: `blockblaz/zeam:devnet4` (sha pulled 2026-05-07T09:42:03Z is on `zeam_1` and ran cleanly; same image on `zeam_0` panicked 21 times).
- `zeam_0` started with `--is-aggregator --aggregate-subnet-ids 0,1 --checkpoint-sync-url …` (full cmd in dockerd journal).
- Both zeam nodes were on the same `blockblaz/zeam:devnet4` image. Difference: zeam_0 is the cross-subnet aggregator (`--aggregate-subnet-ids 0,1`), zeam_1 is a regular subnet-1 validator. The aggregator role exercises the gossip-publish hot path much more frequently (subnet 0 + subnet 1 + aggregation topic), which is what `delay_map` is timing.
- Other zeam-the-aggregator hosts on this devnet: ethlambda_1 also runs `--aggregate-subnet-ids 0,1` but ethlambda is a Rust client (not zeam), so it doesn't surface this bug.

## Asks

1. **Reproduce locally** — run a single-node zeam with `--is-aggregator --aggregate-subnet-ids 0,1` against a busy gossipsub network and `RUST_BACKTRACE=full` to get the application call frames into `delay_map::HashMapDelay::insert` / similar.
2. **Audit `delay_map` usage** in the rust-bridge — every place that calls `insert` followed by later `remove`/`update` on the same key needs to handle the race where the key has expired and been popped between the two calls. Likely candidates:
- gossipsub mesh peer expirations
- discv5 query timeouts
- reqresp pending-stream timeouts
3. **Bump or replace `delay_map`** — `0.4.1` is current (used by sigp/lighthouse). Check if a newer version fixes the desync, or migrate the affected sites to `tokio_util::time::DelayQueue` directly with explicit pre-removal-check semantics.
4. **Convert the abort to a recoverable error** — for the rust-bridge, an isolated `delay_map` desync should not take down the whole zeam process. The thread that owns the gossipsub/discv5 service can drop the bad entry, log, and continue.

## Cross-reference

- #698 — *Panic on checkpoint-sync restart: 'no reactor running' in delay_map (Tokio runtime context missing)* — same crate (`delay_map`), different error (no Tokio reactor vs. queue/HashMap desync). Root-cause likely shares the same call-site family but the surface symptom is different.
- #788 — *zeam_0: fork choice resets to finalized checkpoint under timing desync (FutureSlot rejection loop)* — describes a similar **end state** (zeam_0 wedged, head reset to finalized) but the **trigger** there was clock desync. Today the trigger is the delay_map panic; the fc_initing wedge is the downstream consequence.

## Logs

Full panic block plus the surrounding 4 lines on zeam_0 (Loki, 2026-05-07 10:13–10:15 UTC):

```
May-07 10:14:34.562 [s=555 i=1] [info] (zeam): [network] network-0:: Peer disconnected: …(qlean_1) direction=inbound reason=timeout
May-07 10:14:34.562 [s=555 i=1] [info] (zeam): [node] peer disconnected: …(qlean_1), direction=inbound, reason=timeout, total peers: 14
May-07 10:14:34.562 [s=555 i=1] [info] (zeam): [network] rust-bridge: Scheduling reconnection to peer /ip4/37.27.250.20/udp/9001/quic-v1/p2p/… (attempt 1/5) in 5s
May-07 10:14:34.600 [s=555 i=1] [info] (zeam): [forkchoice] slot_interval=1 duration=0.800s
… (panic at 10:14:53) …
thread '' (1) panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/delay_map-0.4.1/src/hashmap_delay.rs:94:46:
fatal runtime error: failed to initiate panic, error 5, aborting
…
```

Loki query strings used for full reproduction:

```
{container="zeam_0"} |~ "panicked at /home/runner/.cargo/registry/src/.*delay_map"
{container="zeam_0"} |~ "forkchoice still initing"
{container="zeam_1"} |~ "panicked at" ← returns 0 across the same window
```

Loki: `http://46.225.10.32:3100/`

Start with delay_map-0.4.1/src/hashmap_delay.rs:94:46 and audit the rust-bridge call sites named in the issue, especially gossipsub, discv5, and reqresp timeout paths. Reproduce with --is-aggregator, --aggregate-subnet-ids 0,1, and RUST_BACKTRACE=full on a busy gossipsub network. Done means the desync is reproduced or explained, affected usage is addressed, and an isolated failure no longer crashes or wedges zeam.

Written by the indexing model from the issue text.