ethereum-optimism / ethereum-optimism/optimism
op-supernode: fresh virtual node has zero FinalizedL1 for one epoch-poll interval, publishing the interop activation anchor as finalized (poisons EL label, forces 44k-block safe rewind)
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 134
Description
> **Correction, edited after filing:** the first version of this issue blamed `verifiedDB.Rewind` for emptying the verified index. That was wrong — `Rewind` deletes entries *at or after* a timestamp, so older entries survive and the downward scan in `VerifiedBlockAtL1` would have found one. The actual trigger is a **zero-valued `FinalizedL1`** on a freshly created virtual node, proven below from the L1 finality signal log. The observed symptoms, the amplification, and the EL evidence are unchanged.
## Summary
Every invalidation on `interop-reorg-4` replaces the affected virtual node. A fresh virtual node's `StatusTracker` starts with `FinalizedL1` **zero-valued**, and op-node primes L1 finality only from a ticker with no immediate first poll — so for up to one `L1EpochPollInterval` (**default 384s / 6.4 min**) the node has no L1 finality view at all.
`FinalizedL2Head` passes that zero straight into `VerifiedBlockAtL1`, whose first line treats it as "no verified block" and returns the **interop activation anchor**. Published finalized therefore drops to L2 block `12107` — a ~90,000-block regression — on every chain in the dependency set simultaneously.
It does not stay contained to finalized. The anchor is pushed to the EL in a forkchoice update, **op-reth accepts it as `Valid`**, the EL's `finalized` label regresses, and the *next* reset reads that label as its walkback bound and **rewinds the safe head by 44,389 blocks**. That part is log-confirmed on both sides, not theoretical.
Observed on `interop-reorg-4`, 2026-07-28, chains `420120187` and `420120186`, 4 occurrences in one hour. All three supernodes show it independently with identical per-chain anchor hashes — `supernode-0` vn `d881`, `supernode-1` vn `508b`, `supernode-2` vn `366a`, all at `21:00:45-46` with `finalized=0x7628...:12107` on chain `420120187`.
## Root cause
`FinalizedL2Head` (`op-supernode/supernode/chain_container/super_authority.go`):
```go
ss, err := c.SyncStatus(ctx)
...
contribution, err := c.verifierContribution(v.VerifiedBlockAtL1(c.chainID, ss.FinalizedL1))
```
`VerifiedBlockAtL1` (`op-supernode/supernode/activity/interop/interop.go`):
```go
func (i *Interop) VerifiedBlockAtL1(chainID eth.ChainID, l1Block eth.L1BlockRef) (eth.BlockID, uint64, error) {
if _, ok := i.verifiedDB.LastTimestamp(); !ok && !i.backfillCompleted.Load() {
return eth.BlockID{}, 0, ErrNotStarted // added by a0bbd932a7 (cold start)
}
if l1Block == (eth.L1BlockRef{}) {
return eth.BlockID{}, i.activationCap(), nil // <-- THIS. zero L1 ref -> activation anchor
}
...
```
`ss.FinalizedL1` is only ever written by `StatusTracker.OnL1Finalized` (`op-node/rollup/status/status.go:132`, the sole assignment — verified by grep). `OnL1Finalized` is driven by:
```go
// op-node/node/node.go:373
l1FinalizedSub := eth.PollBlockChanges(node.log, node.l1Source, onL1Finalized, eth.Finalized,
cfg.L1EpochPollInterval, time.Second*10)
```
and `PollBlockChanges` (`op-service/eth/heads.go:89`) is a bare `time.NewTicker(interval)` loop — **no priming poll before the first tick**. `L1EpochPollIntervalFlag` defaults to `time.Second * 12 * 32` = **384s** (`op-node/flags/flags.go:313`).
`StatusTracker` is constructed per driver (`op-node/rollup/driver/driver.go:55`), i.e. per virtual node. So **every** newly created virtual node publishes `FinalizedL1 == eth.L1BlockRef{}` for its first 384 seconds, and during that entire window its published finalized head is the interop activation anchor.
Note this is *not* a hold-previous situation the engine could ride out: `SyncStatus` succeeds and `ss.LocalSafeL2.Time` is populated (L2 heads come from the engine immediately), so the `HoldPrevious` and `PreActivation` branches are both bypassed. The zero is only in the L1 field.
### Why the window lasted 15 minutes rather than 6.4
Virtual node churn re-arms the timer. One vn per chain per supernode, replaced sequentially:
```
21:00:33 info virtual node stopped chain_id=420120187 vn_id=c36d
21:00:41 info Reset of Engine is completed chain_id=420120187 vn_id=d881 <- new vn
21:06:47 info virtual node stopped chain_id=420120187 vn_id=d881
21:06:51 info Loaded current L2 heads chain_id=420120187 vn_id=ca57 <- new vn
```
L1 finality signals actually delivered, both chains (`New L1 finalized block`):
| time (186 / 187) | vn (186 / 187) | `l1_finalized` | Δ |
|---|---|---|---|
| 20:50:46 / 20:50:47 | 93ab / ab9d | 11370960 | |
| 20:58:19 / 20:58:20 | f8eb / c36d | 11370992 | +32, 7m33s |
| **21:13:13 / 21:13:14** | 9b93 / **ca57** | 11371054 | **+62, 14m54s — one signal missing** |
| 21:19:37 / 21:19:38 | 9b93 / ca57 | 11371086 | +32, 6m24s |
| 21:26:01 / 21:26:02 | 9b93 / ca57 | 11371118 | +32, 6m23s |
L1 finality itself advanced normally (+62 ≈ 2 epochs across the gap). The signal is missing because **d881 died 18 seconds before its first tick**:
- d881 created `21:00:41` → first poll due `21:00:41 + 384s = 21:07:05` → stopped `21:06:47`. Never primed. Anchor for its entire 6m06s life.
- ca57 created `21:06:51` → first poll due `21:13:15` → observed `21:13:14`. **383s — matches the interval exactly.**
So the **anchor window** — published finalized = `12107` — ran `21:00:45` → `21:13:14`, about 12.5 minutes, because it spanned two consecutive un-primed vn generations rather than one.
The unsafe-ingestion **gate** is a separate clock and was *not* meaningfully extended here. It clears when `finalized >= maxDeniedHeight`, and the healthy value (`101636`) was itself below `maxDeniedHeight=102109`, so the gate would have been open until real L2 finality passed `102109` either way. It closed at `21:16:02`, just after the first primed signal. In general the anchor can extend the gate by at most one priming window (384s) — the case where real finality has passed the denied height but the node is reading `12107` — but in this incident the two clocks expired together and the extension was ~0.
## Observed: finalized head
Healthy before the invalidation, `Verified` source live:
```
20:58:19 warn super authority finalized a block ahead of local finalized; using local finalized
chain_id=420120187 vn_id=c36d
super_authority_finalized=0xa9d4...:101636 local_finalized=0xd1c7...:101176
```
Reset creates d881, which loads the correct finalized from the EL:
```
21:00:41 info Reset of Engine is completed chain_id=420120187 vn_id=d881
local_unsafe=0x3de9...:102231 cross_unsafe=0x3de9...:102231
local_safe=0xa9d4...:101636 cross_safe=0xa9d4...:101636 finalized=0xa9d4...:101636
```
4 seconds later, published finalized is the anchor:
```
21:00:45 warn Gating unsafe ingestion during invalidation recovery
chain_id=420120187 vn_id=d881 maxDeniedHeight=102109
finalized=0x7628dfd671d42a82e505b2d323552117cef30db20faec3fadf174756010af45a:12107
```
Pinned there for d881's whole life. `Sync progress` logs `FinalizedHead()`/`SafeL2Head()` directly (`engine_controller.go:570-571`), so these are the published values:
```
21:05:22 info Sync progress chain_id=420120187 vn_id=d881 reason="consolidated block with L1"
l2_finalized=0x7628...:12107 <-- anchor, pinned, identical hash throughout
l2_safe=0x47a7...:102737 <-- safe unaffected, advancing
l2_pending_safe=...:102751 l2_unsafe=...:102751
21:05:24 info Sync progress ... l2_finalized=0x7628...:12107 l2_safe=0x10b2...:102822 l2_unsafe=...:102837
```
Recovery, after ca57 finally receives L1 finality at `21:13:14`:
```
21:16:02 warn Resuming unsafe ingestion, finality passed the invalidation
chain_id=420120187 vn_id=ca57 maxDeniedHeight=102109 finalized=0xefa3...:102109
```
**Both chains, same height, different hashes** — the tell that the value is computed from a timestamp, not read from a DB:
```
21:00:45 warn Gating ... chain_id=420120186 vn_id=b49c maxDeniedHeight=102223 finalized=0x0229...:12107
21:00:45 warn Gating ... chain_id=420120187 vn_id=d881 maxDeniedHeight=102109 finalized=0x7628...:12107
```
Earlier occurrences the same hour: `20:06:0x` and `20:52:0x`, both chains, always `...:12107`.
Anchor arithmetic (derived, self-consistent across three independent `Sync progress` lines): `l2_time - l2_unsafe.Number` = `1785271820-102128` = `1785272443-102751` = `1785271818-102126` = **1785169692**, i.e. 1s blocks and L2 genesis time `1785169692`. Block `12107` ⇒ ts `1785181799` ⇒ `activationTimestamp = 1785181800`. Exactly `activationCap()` = `activationTimestamp - 1`, and `TargetBlockNumber(1785181799) = 12107`.
## Why safe is not hit directly
The two SuperAuthority accessors differ in whether they take an L1 bound at all:
| | accessor | verifier call | with `FinalizedL1 == zero` |
|---|---|---|---|
| safe | `FullyVerifiedL2Head` | `LatestVerifiedL2Block(chainID)` — **no L1 argument** | unaffected → `Verified` → safe fine |
| finalized | `FinalizedL2Head` | `VerifiedBlockAtL1(chainID, ss.FinalizedL1)` | zero ref → `Anchor` |
Safe cannot see the zero because its query has no L1 parameter. That asymmetry is what makes the incident diagnosable, and it is why `l2_safe=102737` and `l2_finalized=12107` appear on the same log line.
## Observed: safe head, first reset — rewinds to finalized, below the last cross-safe
```
21:00:33 info Sync progress chain_id=420120187 vn_id=c36d
l2_finalized=0xa9d4...:101636 l2_safe=0xa34b...:102220 l2_unsafe=...:102230
21:00:33 info Sync progress chain_id=420120187 vn_id=c36d
l2_finalized=0xa9d4...:101636 l2_safe=0xa9d4...:101636 l2_unsafe=...:102231
^^^^^^ 102220 -> 101636 == the finalized head
21:00:41 info Resetting safe head db chain_id=420120187 vn_id=d881 l2=0xa9d4...:101636
```
Safe lands on `101636`, the **finalized** head — **584 blocks below the last cross-safe (`102220`)**, and 473 below the invalidation point (`maxDeniedHeight=102109`, so `102108` is the deepest rewind correctness requires).
Mechanism is `FindL2Heads` (`op-node/rollup/sync/start.go:282-288`): the walkback from unsafe searches for a safe candidate with a full sequence window of canonical L1 origins, and if it reaches the finalized head first it hard-assigns safe to it:
```go
// Don't traverse further than the finalized head to find a safe head
if n.Number == result.Finalized.Number {
lgr.Info("Hit finalized L2 head, returning immediately", ...)
result.Safe = n
return result, nil
}
```
```
21:00:41 info Hit finalized L2 head, returning immediately chain_id=420120187 vn_id=d881
unsafe=0x3de9...:102231 safe=0xfd6b...:102222 finalized=0xa9d4...:101636
```
So safe-rewind depth is set by **wherever the finalized head happens to be**, not by the invalidation point. Here finalized was still healthy, so the overshoot was only ~500 blocks. That coupling is what makes the anchor catastrophic on the next reset.
## Amplification: the anchor poisons the EL finalized label → 44,389-block safe rewind
`FindL2Heads` takes its bound from `currentHeads`, which reads the **EL's `finalized` label** (`start.go:78`, `L2BlockRefByLabel(eth.Finalized)`) — not `localFinalizedHead`. And the engine pushes the anchor to the EL via `fc.FinalizedBlockHash = e.FinalizedHead().Hash` (`engine_controller.go:694`, `tryUpdateEngineInternal`, on the derivation path which was live throughout — `Inserted new L2 unsafe block` 102364+ at `21:00:58`).
op-reth accepts it. Chain `420120186`, anchor block `12107` = `0x022906f9...` (hash matches the gating log above exactly):
```
21:06:47 info target=reth_node_events::node message="Forkchoice updated"
head_block_hash=0x084ebbcd... safe_block_hash=0x5fc39a4b...
finalized_block_hash=0x022906f9f37efad1a5f15fb810b85b6837fca06d3a903cddc085fa8b284f6fab
21:06:47 warn target=engine::tree forkchoice_status: Valid ... finalized_block_hash: 0x022906f9...
```
Four seconds later the next reset on chain `420120187` reads the poisoned label back out of the EL:
```
21:06:51 info Loaded current L2 heads chain_id=420120187 vn_id=ca57
unsafe=0xd3e2...:102966 safe=0x3d36...:102826
finalized=0x7628dfd671d42a82e505b2d323552117cef30db20faec3fadf174756010af45a:12107
^^^^^^^^^^^^ EL finalized label is now the anchor (was 101636)
```
With the floor at `12107` instead of `101636`, the `n.Number == result.Finalized.Number` early return never fires, so the walkback keeps descending until the sequence-window condition at `start.go:277-280` is satisfied and it exits via `if ready { result.Safe = n }` (`start.go:322`):
```
21:07:35 info Reset of Engine is completed chain_id=420120187 vn_id=ca57
local_unsafe=0xd3e2...:102966 cross_unsafe=0xd3e2...:102966
local_safe=0x7eeb...:58437 cross_safe=0x7eeb...:58437 finalized=0x7628...:12107
```
**Safe: `102826` → `58437`. A 44,389-block rewind**, requiring re-derivation of ~44k blocks.
The stopping point is quantitatively consistent with the sequence-window exit: the walkback descended `102966 → 58437` = 44,529 L2 blocks; at 1s L2 blocks and 12s L1 blocks that is ~3,711 L1 blocks of origin descent, and `SyncLookback()` returns `SeqWindowSize` (`op-node/rollup/types.go:804-811`). Wall-clock also checks out: `21:06:51 → 21:07:35` = 44s for ~44.5k single-block iterations.
Corroborating that the EL label really regressed, a later reset loads `local_finalized` far below the pre-incident value as finality re-climbs:
```
21:16:01 warn super authority finalized a block ahead of local finalized; using local finalized
chain_id=420120187 vn_id=ca57
super_authority_finalized=0xefa3...:102109 local_finalized=0x66cd...:93677
```
Full loop:
1. invalidation → vn replaced → fresh `StatusTracker`, `FinalizedL1 == zero` for up to 384s
2. `VerifiedBlockAtL1(chain, zero)` → `activationCap()` → `Anchor` → published finalized = `12107`
3. FCU sends `finalized_block_hash` = block `12107`; **op-reth returns `Valid`**; EL label regresses
4. next reset's `currentHeads` reads `12107` from the EL → walkback floor collapses → **safe rewinds ~44k blocks**
5. mass re-derivation → more resets → each new vn re-arms another 384s blind window (plausibly the engine behind #21948)
Two defects here beyond the anchor itself: the EL accepting a **backwards** `finalized_block_hash` as `Valid`, and op-node **round-tripping its own finality through the EL label** so one bad publish becomes durable input.
## Blast radius
- **44,389-block safe rewind** on the following reset, via the poisoned EL label. Worst consequence; forces mass re-derivation.
- **Engine API — backwards finalized, delivered and accepted.** op-reth was told `101636`, then `12107`, and answered `Valid`.
- **Unsafe-ingestion gate reads a meaningless input**, though it was not materially extended in this incident. `unsafeDenyGatingActive` gates on `maxDenied > FinalizedHead()`; the healthy value (`101636`) was also below `maxDenied=102109`, so the gate was inherently open until real finality passed the denied height. Worth fixing for correctness — the anchor can extend the gate by up to one priming window (384s) whenever real finality has already passed the denial — not as an availability win.
- **Every virtual node restart is exposed**, not just invalidations — any path that replaces a vn publishes the activation anchor as finalized for up to 384s. Invalidation churn is simply the most frequent trigger.
- **Dashboards/metrics** show finalized falling off a cliff on every rollback, training operators to ignore a real alarm.
- **Additional unexercised path:** `crossSafeFallback` also floors at `FinalizedHead()`, so a concurrent verifier read failure would drag cross-safe to the anchor directly, without the EL round-trip. Did not fire in this window.
- **Latent:** the same unvalidated value feeds `unsafeHead.Number < FinalizedHead().Number` → `CriticalErrorEvent` → node exit (`engine_controller.go:683`). Hard to trigger, since the Anchor branch is ceilinged at `localFinalizedHead`.
## Proposed fix
1. **Prime L1 finality at virtual-node startup.** Fetch `L1BlockRefByLabel(eth.Finalized)` once during init, or give `PollBlockChanges` an immediate first poll before entering its ticker loop (`op-service/eth/heads.go:89`). This closes the window at the source and also fixes L1 *safe* (`l1SafeSub`, same helper, same gap). Cheapest correct fix.
2. **Treat a zero `l1Block` as unknown, not as "nothing verified."** `VerifiedBlockAtL1` should return `ErrNotStarted`/hold-previous when `l1Block == (eth.L1BlockRef{})`, so the engine's existing cache path handles it instead of publishing the anchor. Note a0bbd932a7 already added an `ErrNotStarted` guard on the line *immediately above* this branch for the cold-start case — the zero-L1 case was simply not considered.
3. **Decouple the deny gate:** compare `maxDeniedHeight` against `localFinalizedHead`, which never depended on the verifier or on L1 finality signals and was correct throughout.
4. **Stop round-tripping own finality through the EL label.** Bound the `FindL2Heads` walkback on `localFinalizedHead` rather than `L2BlockRefByLabel(eth.Finalized)` (`start.go:78`), so a bad publish cannot become durable input to the next reset. This is what turns a transient 384s glitch into a 44k-block rewind.
5. **Worth raising upstream:** op-reth answered `Valid` to a forkchoice update whose `finalized_block_hash` moved *backwards* ~90k blocks. Rejecting non-monotonic finalized would contain this at the EL boundary.
(1) or (2) removes the regression; (4) is what prevents any future bad publish from causing a 44k-block rewind.
## Reproduction
Any restart or invalidation-driven replacement of a virtual node in SuperAuthority mode. `interop-reorg-*` reproduces it on every rollback — 4x in the hour examined, on all three supernodes. Tells:
- published finalized equals the same L2 height on *every* chain in the dependency set, with *different* hashes, immediately after a `vn_id` change
- no `New L1 finalized block` log for the new `vn_id` yet
- clears within one `L1EpochPollInterval` of that vn's start
Logs: `gcx logs query -d grafanacloud-logs '{namespace="an-interop-reorg-4-supernode-0"}' --from 2026-07-28T20:00:00Z --to 2026-07-28T22:00:00Z`
## Relevant code
- `op-supernode/supernode/activity/interop/interop.go` — `VerifiedBlockAtL1` (zero-`l1Block` branch), `LatestVerifiedL2Block`, `activationCap`
- `op-supernode/supernode/chain_container/super_authority.go` — `FinalizedL2Head`, `FullyVerifiedL2Head`, `verifierContribution`
- `op-node/rollup/status/status.go:132` — `OnL1Finalized`, sole writer of `FinalizedL1`
- `op-node/node/node.go:373` + `op-service/eth/heads.go:89` — un-primed ticker
- `op-node/flags/flags.go:313` — `L1EpochPollInterval` default 384s
- `op-node/rollup/driver/driver.go:55` — per-vn `StatusTracker`
- `op-node/rollup/sync/start.go` — `currentHeads:78`, finalized floor `:282-288`, sequence-window exit `:277-280`/`:322`
- `op-node/rollup/engine/engine_controller.go` — `FinalizedHead:328`, `resolveAnchorAsFinalized:378`, `crossSafeFallback:296`, `unsafeDenyGatingActive:772`, FCU `:694`, node-exit check `:683`
## Related
- #21097 — structural root cause of this bug class (SuperAuthority head accessors, anchor fallback, zero-value sentinels). This is an uncovered instance: same function, adjacent branch.
- #21092 — cold-start variant of the anchor pinning.
- #21948 — invalidation recovery repeatedly resetting; each new vn re-arms a 384s anchor window, so the two likely compound.
Contributor guide
Research direction
Start with op-node/rollup/status/status.go, op-service/eth/heads.go, and op-supernode/supernode/chain_container/super_authority.go to trace how a new virtual node obtains FinalizedL1. Then inspect VerifiedBlockAtL1 in op-supernode/supernode/activity/interop/interop.go and the finalized forkchoice path in engine_controller.go. Done should prevent a zero L1 finality value from publishing the activation anchor or regressing the EL finalized label.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100