op-supernode/op-reth: CL sync recovery can stall when SafeDB/bookkeeping references EL-missing block

## Summary

While recovering `bn-interop-jnt-v2` supernode 3 in CL sync mode, chain `420120103` stalled because op-supernode/op-node bookkeeping believed L2 block `47725` existed, but the paired op-reth no longer served that payload. A plain supernode restart cleared the stale reference, but then op-reth's Engine API hung on `engine_forkchoiceUpdatedV3` until reth itself was restarted.

This was observed during a CL-sync-from-genesis recovery of SN3, after wiping only SN3 state and updating it to the latest tagged rc images available to us. No SN0/SN1/SN2 DBs were wiped as part of this particular repro.

## Environment

Network: `bn-interop-jnt-v2`

Supernode: `sn-3`

Namespaces:

- supernode: `bn-interop-jnt-v2-supernode-3`
- chain2 EL: `bn-interop-jnt-v2-1-ops-reth-a-sn-3`

Chain:

- chain ID: `420120103`
- recovery mode: CL sync

Images / versions:

- `op-supernode`: `us-docker.pkg.dev/oplabs-tools-artifacts/images/op-supernode:v0.3.1-rc.2@sha256:9f18ebcb63ccc2d85845daaf22f187cd56a7d465f8660a522ccfd33524f39e17`
- runtime log: `supernode starting version=v0.3.1-rc.2`
- `op-reth`: `us-docker.pkg.dev/oplabs-tools-artifacts/images/op-reth:v2.2.5@sha256:42dbab6276aa795bb9ca4e94c01e8baed8b6f5d9159d1fa12e352630814e62c6`
- runtime log: `Starting Reth version="2.2.0-dev (81c0261)"`

Note: `op-supernode:v0.3.1-rc.2` likely does not contain https://github.com/ethereum-optimism/optimism/pull/21049 if that PR has not yet landed in the tagged image.

## Timeline / Evidence

### 1. CL sync recovery hit an EL-missing block

Before restart, SN3 chain2 reth served:

```text
latest: 47724 / 0x660f71a246051b72f2754dda0f3a6786279698abc362f3e74839e64d60362ef7
safe: 15071 / 0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b
finalized: 15071 / 0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b
block 47725: null
```

But op-supernode/op-node state for chain2 was trying to derive from block `47725`:

```text
47725 / 0xcffdf51258baa5d011e207dbd07d180476d84a9961bacaccd00169d2b61f395c
```

The live loop then repeatedly failed with:

```text
t=2026-05-27T20:16:26+0000 lvl=warn msg="Engine temporary error" chain_id=420120103 vn_id=63c2 err="derivation failed: temp: failed to retrieve L2 parent block: failed to determine block-hash of hash 0xcffdf51258baa5d011e207dbd07d180476d84a9961bacaccd00169d2b61f395c, could not get payload: not found"
```

Earlier logs show why this stale reference was plausible. Reth had previously observed/canonicalized `47725`, then later no longer served it:

```text
# around 2026-05-27T19:44:40Z
Block added to canonical chain number=47725 hash=0xcffdf51258baa5d011e207dbd07d180476d84a9961bacaccd00169d2b61f395c
Received new payload number=47725 hash=0xcffdf51258baa5d011e207dbd07d180476d84a9961bacaccd00169d2b61f395c

# around 2026-05-27T19:44:50Z
msg="Failed to share forkchoice-updated signal" chain_id=420120103 state="...HeadBlockHash:0xcffdf51258baa5d011e207dbd07d180476d84a9961bacaccd00169d2b61f395c..." err="context deadline exceeded"

# around 2026-05-27T19:45:00Z
msg="Record local safe head" chain_id=420120103 l2=0xcffdf51258baa5d011e207dbd07d180476d84a9961bacaccd00169d2b61f395c:47725
msg="Engine temporary error" chain_id=420120103 err="...could not get payload: not found"
```

Interpretation: during CL sync recovery, block `47725` became part of op-supernode/op-node local-safe bookkeeping, but the paired EL later could not answer for that block. After that, derivation got pinned on `could not get payload: not found`.

### 2. Plain op-supernode restart cleared the stale 47725 reference

We restarted only the SN3 supernode pod:

```text
kubectl delete pod op-supernode-0 -n bn-interop-jnt-v2-supernode-3
```

No DB/PVC wipe was performed for this restart.

After restart, supernode no longer tried to use missing block `47725`. Chain2 status moved back to EL state that reth could serve:

```text
unsafe_l2: 47724 / 0x660f71a246051b72f2754dda0f3a6786279698abc362f3e74839e64d60362ef7
local_safe_l2: 15071 / 0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b
safe_l2: 15071 / 0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b
finalized_l2: 15071 / 0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b
```

This cleared the `could not get payload` loop, but recovery still did not progress because Engine API FCU calls began timing out.

### 3. Reth Engine API accepted connections but hung on FCU

The repeated supernode/op-node error after the supernode restart was:

```text
t=2026-05-27T20:36:21+0000 lvl=warn msg="Failed to share forkchoice-updated signal" chain_id=420120103 vn_id=75ac state="&{HeadBlockHash:0x660f71a246051b72f2754dda0f3a6786279698abc362f3e74839e64d60362ef7 SafeBlockHash:0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b FinalizedBlockHash:0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b}" err="Post \"http://execution-engine.bn-interop-jnt-v2-1-ops-reth-a-sn-3.svc.cluster.local:8551\": context deadline exceeded"

t=2026-05-27T20:36:21+0000 lvl=warn msg="Engine temporary error" chain_id=420120103 vn_id=75ac err="temp: failed to sync forkchoice with engine: Post \"http://execution-engine.bn-interop-jnt-v2-1-ops-reth-a-sn-3.svc.cluster.local:8551\": context deadline exceeded"
```

The Engine API port itself was reachable: unauthenticated `engine_exchangeCapabilities` returned `401 Unauthorized` immediately.

JWTs matched between supernode and reth by direct in-cluster comparison. The secret value is intentionally omitted.

But a manual authenticated `engine_forkchoiceUpdatedV3` to reth with the same head/safe/finalized hashes timed out after 15s with no response:

```text
head: 0x660f71a246051b72f2754dda0f3a6786279698abc362f3e74839e64d60362ef7
safe: 0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b
finalized: 0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b

curl: (28) Operation timed out after 15006 milliseconds with 0 bytes received
HTTP_CODE=000 TOTAL_TIME=15.006482
```

Reth public RPC remained alive at the same time and returned latest/safe/finalized, so this looked like an Engine API / FCU handling hang rather than a service/DNS/JWT outage.

### 4. Plain reth restart unblocked FCU and recovery

We then restarted only SN3 chain2 reth:

```text
kubectl delete pod op-reth-0 -n bn-interop-jnt-v2-1-ops-reth-a-sn-3
```

No DB/PVC wipe was performed for this restart.

On startup, reth spent about 95s loading ExEx WAL before the Engine API started:

```text
ts=2026-05-27T20:44:49.567496518Z level=info target=reth::cli message="Starting Reth" version="\"2.2.0-dev (81c0261)\""
ts=2026-05-27T20:44:50.766618337Z level=info target=reth::cli message="Loading ExEx Write-Ahead Log..."
ts=2026-05-27T20:46:24.777627253Z level=info target=reth::cli message="ExEx Manager started"
ts=2026-05-27T20:46:24.781068632Z level=info target=reth::cli message="Engine API handler initialized"
ts=2026-05-27T20:46:24.783416923Z level=info target=reth::cli message="RPC auth server started" url=0.0.0.0:8551
```

Immediately after that, reth accepted FCU and recommitted the known canonical head:

```text
ts=2026-05-27T20:46:25.232639746Z level=info target=reth_node_events::node message="Received forkchoice updated message when syncing" head_block_hash=0x660f71a246051b72f2754dda0f3a6786279698abc362f3e74839e64d60362ef7 safe_block_hash=0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b finalized_block_hash=0xec69b3a02307eadc405bd2e20c1cf74d9ff28882ed068f65118107e2756b227b

ts=2026-05-27T20:46:25.36577321Z level=info target=reth_node_events::node message="Block added to canonical chain" number=47723 hash=0x348277...
ts=2026-05-27T20:46:25.370152528Z level=info target=reth_node_events::node message="Block added to canonical chain" number=47724 hash=0x660f71...
ts=2026-05-27T20:46:25.370384433Z level=info target=reth_node_events::node message="Canonical chain committed" number=47724 hash=0x660f71...
```

Then op-supernode/op-node walked back the L1 cursor and resumed local-safe/safe progress:

```text
t=2026-05-27T20:47:52+0000 lvl=info msg="Walking back L1Block by hash" chain_id=420120103 vn_id=75ac curr=0xcf7e58...:10841049 next=0x43d1ef...:10841048 l2block=0xe7fde3...:15263
...
t=2026-05-27T20:47:53+0000 lvl=info msg="Walking back L1Block by hash" chain_id=420120103 vn_id=75ac curr=0x4f85ee...:10841038 next=0xc1fbce...:10841037 l2block=0xbf8eb8...:15083
```

Recovery signal:

```text
2026-05-27T20:50:19Z current_l1=10841257 unsafe=47722 local_safe=15095 pending_safe=15095 safe=15095 finalized=15071
2026-05-27T20:50:47Z current_l1=10841261 unsafe=47722 local_safe=16823 pending_safe=16823 safe=16823 finalized=16531
```

And logs:

```text
t=2026-05-27T20:50:21+0000 lvl=info msg="Record local safe head" chain_id=420120103 vn_id=75ac l2=0x492c6636...:15135 l1=0x738dad57...:10841257
...
t=2026-05-27T20:50:49+0000 lvl=info msg="Record local safe head" chain_id=420120103 vn_id=75ac l2=0x49f8dd2c...:16851 l1=0x39bcfe2f...:10841261
```

## Actual behavior

During CL sync recovery, op-supernode/op-node could get pinned on a local-safe/unsafe bookkeeping reference to an L2 block that reth did not serve anymore:

```text
could not get payload: not found
```

A plain supernode restart dropped that stale reference, but then reth's Engine API repeatedly timed out on FCU despite RPC being reachable and JWTs matching. A plain reth restart fixed the FCU hang and allowed CL-sync recovery to resume.

## Expected behavior

- op-supernode/op-node should not keep retrying forever against a local-safe/unsafe L2 block that the EL cannot serve.
- If the EL loses or no longer serves a block that op-supernode/op-node has recorded as local safe during CL sync recovery, the node should be able to invalidate/reconcile that bookkeeping without requiring a process restart.
- reth should not hang indefinitely on an FCU for a head/safe/finalized tuple it can serve over public RPC. It should either return a valid FCU response or a concrete error.

## Suspected related areas

- Supernode/op-node safe/finalized bookkeeping during CL sync recovery, especially around failed EL reads and whether failed verifier/EL lookups are cached as valid progress.
- Potentially related to https://github.com/ethereum-optimism/optimism/pull/21049, if `op-supernode:v0.3.1-rc.2` does not include that change.
- op-reth Engine API FCU handling after a block was previously accepted/canonicalized but later no longer available via payload lookup.

## Repro shape

This is not yet a minimal unit repro, but the observed live repro shape was:

1. Run a supernode verifier in CL sync mode against an op-reth archive EL.
2. During recovery, have the EL briefly accept/canonicalize an L2 block `N`.
3. Have op-supernode/op-node record `N` as local safe/unsafe bookkeeping.
4. Later, have the EL no longer serve `N` by hash/number.
5. Observe derivation loop pinned on `could not get payload: not found`.
6. Restart only op-supernode: stale reference clears, but FCU may then hang against reth.
7. Restart only reth: FCU succeeds and CL sync recovery resumes.