Caplin (gloas/ePBS): forkchoice head advances only per-epoch at tip — EL head lags ~1 epoch

## Summary

On gloas/ePBS (`glamsterdam-devnet-6`), a synced erigon + embedded-Caplin node cannot reach true chain tip. It settles ~1 epoch behind: the forkchoice head advances only at epoch boundaries, so the EL is executed/committed once per epoch and JSON-RPC lags ~1 epoch (~6 min). On pre-gloas mainnet the same node follows one block per slot at tip.

## Observed

- Blocks arrive every slot (`OnBlock` per ~12s), added to the fork graph.
- CL forkchoice head is pinned to epoch boundaries: `head_slot` = 242400, later 243456 — both exact multiples of 32 — each frozen for a full epoch while the current slot advances.
- EL follows per epoch: `parallel executed blks=26–32`, `Timings: Forkchoice flush+commit` exactly 384s (= 32 slots) apart, never `blks=1`. Steady ~1.5-epoch (~48-block) lag. `is_syncing=false`.

## Root cause

**1. The gloas head walk only advances through *verified* payloads.** `computeHeadGloas`/`getNodeChildren` (`cl/phase1/forkchoice/get_head.go`, `payload_vote.go`) walks from the justified checkpoint; to step from block N to its child N+1 it must first reach the `FULL(N)` node, which is only offered when `IsPayloadVerified(N)` is true. If sub-epoch payloads aren't verified, the head stops at the justified block.

**2. The per-slot path verifies only on a *synchronous* EL-Valid; a behind node never gets one.** The per-slot gossip path `on_execution_payload.go` (`validatePayloadWithEL`) *does* mark the payload verified when `NewPayload` returns `PayloadStatusValidated` (line 306 → `markPayloadVerifiedLocked`) — so a caught-up node follows per slot. But when the node is behind, `NewPayload` returns `PayloadStatusNone` ("EL behind": the EL hasn't executed parent N-1 yet), and that branch *defers* — persists the envelope, queues it in `pendingELPayloads`, `AddOptimisticCandidate` — **without** marking verified. Only the batched `chain_tip_sync` path (`drainPendingGloasPayloads`, `verifyUnverifiedGloasPayloads`) later re-submits and marks verified.

**3. A hard 1-block-per-cycle ceiling makes the gap un-closeable — it's a circular dependency, not just a slow drain.** The integrated EL (`ExecutionClientDirect.NewPayload`) only *validates* a payload that is exactly `EL_head + 1`:

```go
headHeader := cc.chainRW.CurrentHeader(ctx)
if headHeader == nil || header.Number > headHeader.Number+1 {
return PayloadStatusNotValidated, nil // more than 1 ahead → stored, not executed
}
```

and the EL's `CurrentHeader` only advances via `ForkChoiceUpdate`, which Caplin issues once per clstages cycle (`stages/forkchoice.go`) targeting the forkchoice head. So per cycle: `drainPendingGloasPayloads` drains the whole backlog synchronously, but only the single `EL_head+1` payload validates; every deeper one returns `NotValidated` and requeues. That one verification lets the forkchoice head advance by one, the next cycle's FCU moves `CurrentHeader` by one, and the next drain verifies one more. **Net: exactly one block verified per cycle ≈ chain rate**, so a gap opened once (e.g. during the earlier PeerDAS stall / checkpoint sync) never closes — a stable ~1-epoch-behind equilibrium.

The circularity: verify(N) needs `EL_head == N-1`; `EL_head` advances by FCU to the forkchoice head; the forkchoice head advances only through verify(N). Each cycle breaks exactly one link forward. A caught-up node never hits it (every gossiped payload is exactly `EL_head+1`, validates immediately, head tracks per slot — the mainnet behavior); only a node carrying a gap does. (The `maxGloasVerificationSweepPerCycle = 32` sweep is **not** the driver — it never runs here: zero `GLOAS verification sweep` log lines, `swept=0`.)

Corroborating symptom: PTC attestations for these blocks are rejected — `bad attestation received err="PTC attestation requires verified payload envelope"` — so the sub-epoch blocks are also denied payload weight.

## Impact

- Node never reaches true tip on gloas; committed EL head / JSON-RPC stale by up to ~1 epoch.
- Stable ~1-epoch-behind equilibrium (drains at chain-rate each epoch, never closes the gap).

## Spec + reference-client context

**Spec** (`specs/gloas/fork-choice.md`, v1.7.0-alpha.12): `get_node_children` emits the `FULL(N)` child only `if is_payload_verified(store, N)`, and `is_payload_verified(root) := root in store.payloads`, written by `on_execution_payload_envelope` per slot. A synced node's head is expected to track within ~1 slot; per-epoch advancement is a symptom of not verifying sub-epoch payloads promptly. The spec's verify step requires a real EL validation, so the marking must follow an EL result — matching Caplin's line-306 behavior for the caught-up case.

**Reference clients handle the behind case so it self-heals:**
- Prysm (`blockchain/receive_execution_payload_envelope.go` → `InsertPayload`) accepts the payload optimistically when the EL returns `SYNCING`, letting the head advance and dragging the EL forward.
- Lighthouse gates its per-slot verify behind a synced EL and re-drives on the next tick.

Caplin has neither: on `PayloadStatusNone` it defers to the batched path and never re-engages the per-slot verify, so it can't climb back to per-slot cadence once behind.

## Fix direction

The per-slot mark-on-Valid already exists and is correct. The missing piece is a **catch-up drive that breaks the one-block-per-cycle ceiling** when the verified head lags — walk the backlog in order and, for each block that is `EL_head+1`, `NewPayload` it, and once it validates, advance the EL (`ForkChoiceUpdate`) so the next block becomes `EL_head+1`, looping until the verified head reaches the seen tip. This keeps the spec invariant (mark verified only after a real EL validation) while letting a lagging node climb back to per-slot cadence in one recovery pass instead of one block per cycle.

Alternatives considered: re-entering `ForwardSync` — but it catches up via P2P range-download and doesn't re-drive blocks already in the graph, and its transition gate (`MetaCatchingUp`) keys off `HighestSeen()` (gossip-seen, always at tip) not the verified-head lag, so it never fires here anyway. Prysm-style optimistic acceptance of `NotValidated`/`None` would also advance the head but relaxes the spec's "verify only after real EL validation" invariant.

## Environment

`glamsterdam-devnet-6`, erigon + embedded Caplin, recent `main`.

---

## Update: devnet test of a catch-up fix — inert, and a second blocker surfaced

A draft fix for the ceiling (an FCU-interleaved catch-up drive in `chain_tip_sync.go`: after each per-slot payload validates, `ForkChoiceUpdate` the EL to it so the next block becomes `EL_head+1` in the same pass) was built and run on `glamsterdam-devnet-6`. Result: **inert — zero catch-up verifications.** It could not engage, which exposed a distinct, dominant blocker:

- After a node restart, the block just above the EL head is missing its execution-payload envelope (`[chainTipSync] envelope recovery: fetching missing envelopes`). The catch-up walk only collects blocks that already have an envelope, so `EL_head+1` isn't in the set, the oldest candidate is `> EL_head+1`, `NewPayload` returns `NotValidated`, and the drive stops immediately.
- When such an envelope is (re)fetched during forward sync, it sometimes fails EL validation with `cannot derive rlp header: mismatching hash: != ` and the payload is marked invalid. This is a per-payload hashing/pairing mismatch (independent of EL head or the catch-up FCU) — it looks like a forward-sync envelope re-fetch/pairing problem at the gloas boundary, and did **not** occur on the continuously-running baseline.

So there are two separate problems:

- **(A) The FCU one-block-per-cycle ceiling** (this issue's main body) — the steady-state cause of the ~1-epoch lag when envelopes are present, as seen on the original long run (`blks=29` per-epoch bursts).
- **(B) Post-restart envelope availability + `mismatching hash` invalidations** at the boundary — missing/mismatched envelopes for blocks just above the EL head, which stall the node outright and mask (A).

The ceiling fix targets (A) but could not be validated because the restart used to test it put the node into state (B); a clean resync (fresh chaindata + Caplin) would be needed to exercise (A) in isolation. Blocker (B) appears to be its own forward-sync/boundary bug and is the more immediate stall.