ethereum-optimism / ethereum-optimism/optimism

op-supernode: fresh virtual node has zero FinalizedL1 for one epoch-poll interval, publishing the interop activation anchor as finalized (poisons EL label, forces 44k-block safe rewind)

Open
#22,127 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 18h
Merged PRs (30d)
134

Description

> **Correction, edited after filing:** the first version of this issue blamed `verifiedDB.Rewind` for emptying the verified index. That was wrong — `Rewind` deletes entries *at or after* a timestamp, so older entries survive and the downward scan in `VerifiedBlockAtL1` would have found one. The actual trigger is a **zero-valued `FinalizedL1`** on a freshly created virtual node, proven below from the L1 finality signal log. The observed symptoms, the amplification, and the EL evidence are unchanged.

## Summary

Every invalidation on `interop-reorg-4` replaces the affected virtual node. A fresh virtual node's `StatusTracker` starts with `FinalizedL1` **zero-valued**, and op-node primes L1 finality only from a ticker with no immediate first poll — so for up to one `L1EpochPollInterval` (**default 384s / 6.4 min**) the node has no L1 finality view at all.

`FinalizedL2Head` passes that zero straight into `VerifiedBlockAtL1`, whose first line treats it as "no verified block" and returns the **interop activation anchor**. Published finalized therefore drops to L2 block `12107` — a ~90,000-block regression — on every chain in the dependency set simultaneously.

It does not stay contained to finalized. The anchor is pushed to the EL in a forkchoice update, **op-reth accepts it as `Valid`**, the EL's `finalized` label regresses, and the *next* reset reads that label as its walkback bound and **rewinds the safe head by 44,389 blocks**. That part is log-confirmed on both sides, not theoretical.

Observed on `interop-reorg-4`, 2026-07-28, chains `420120187` and `420120186`, 4 occurrences in one hour. All three supernodes show it independently with identical per-chain anchor hashes — `supernode-0` vn `d881`, `supernode-1` vn `508b`, `supernode-2` vn `366a`, all at `21:00:45-46` with `finalized=0x7628...:12107` on chain `420120187`.

## Root cause

`FinalizedL2Head` (`op-supernode/supernode/chain_container/super_authority.go`):

```go
ss, err := c.SyncStatus(ctx)
...
contribution, err := c.verifierContribution(v.VerifiedBlockAtL1(c.chainID, ss.FinalizedL1))
```

`VerifiedBlockAtL1` (`op-supernode/supernode/activity/interop/interop.go`):

```go
func (i *Interop) VerifiedBlockAtL1(chainID eth.ChainID, l1Block eth.L1BlockRef) (eth.BlockID, uint64, error) {
if _, ok := i.verifiedDB.LastTimestamp(); !ok && !i.backfillCompleted.Load() {
return eth.BlockID{}, 0, ErrNotStarted // added by a0bbd932a7 (cold start)
}
if l1Block == (eth.L1BlockRef{}) {
return eth.BlockID{}, i.activationCap(), nil // <-- THIS. zero L1 ref -> activation anchor
}
...
```

`ss.FinalizedL1` is only ever written by `StatusTracker.OnL1Finalized` (`op-node/rollup/status/status.go:132`, the sole assignment — verified by grep). `OnL1Finalized` is driven by:

```go
// op-node/node/node.go:373
l1FinalizedSub := eth.PollBlockChanges(node.log, node.l1Source, onL1Finalized, eth.Finalized,
cfg.L1EpochPollInterval, time.Second*10)
```

and `PollBlockChanges` (`op-service/eth/heads.go:89`) is a bare `time.NewTicker(interval)` loop — **no priming poll before the first tick**. `L1EpochPollIntervalFlag` defaults to `time.Second * 12 * 32` = **384s** (`op-node/flags/flags.go:313`).

`StatusTracker` is constructed per driver (`op-node/rollup/driver/driver.go:55`), i.e. per virtual node. So **every** newly created virtual node publishes `FinalizedL1 == eth.L1BlockRef{}` for its first 384 seconds, and during that entire window its published finalized head is the interop activation anchor.

Note this is *not* a hold-previous situation the engine could ride out: `SyncStatus` succeeds and `ss.LocalSafeL2.Time` is populated (L2 heads come from the engine immediately), so the `HoldPrevious` and `PreActivation` branches are both bypassed. The zero is only in the L1 field.

### Why the window lasted 15 minutes rather than 6.4

Virtual node churn re-arms the timer. One vn per chain per supernode, replaced sequentially:

```
21:00:33 info virtual node stopped chain_id=420120187 vn_id=c36d
21:00:41 info Reset of Engine is completed chain_id=420120187 vn_id=d881 <- new vn
21:06:47 info virtual node stopped chain_id=420120187 vn_id=d881
21:06:51 info Loaded current L2 heads chain_id=420120187 vn_id=ca57 <- new vn
```

L1 finality signals actually delivered, both chains (`New L1 finalized block`):

| time (186 / 187) | vn (186 / 187) | `l1_finalized` | Δ |
|---|---|---|---|
| 20:50:46 / 20:50:47 | 93ab / ab9d | 11370960 | |
| 20:58:19 / 20:58:20 | f8eb / c36d | 11370992 | +32, 7m33s |
| **21:13:13 / 21:13:14** | 9b93 / **ca57** | 11371054 | **+62, 14m54s — one signal missing** |
| 21:19:37 / 21:19:38 | 9b93 / ca57 | 11371086 | +32, 6m24s |
| 21:26:01 / 21:26:02 | 9b93 / ca57 | 11371118 | +32, 6m23s |

L1 finality itself advanced normally (+62 ≈ 2 epochs across the gap). The signal is missing because **d881 died 18 seconds before its first tick**:

- d881 created `21:00:41` → first poll due `21:00:41 + 384s = 21:07:05` → stopped `21:06:47`. Never primed. Anchor for its entire 6m06s life.
- ca57 created `21:06:51` → first poll due `21:13:15` → observed `21:13:14`. **383s — matches the interval exactly.**

So the **anchor window** — published finalized = `12107` — ran `21:00:45` → `21:13:14`, about 12.5 minutes, because it spanned two consecutive un-primed vn generations rather than one.

The unsafe-ingestion **gate** is a separate clock and was *not* meaningfully extended here. It clears when `finalized >= maxDeniedHeight`, and the healthy value (`101636`) was itself below `maxDeniedHeight=102109`, so the gate would have been open until real L2 finality passed `102109` either way. It closed at `21:16:02`, just after the first primed signal. In general the anchor can extend the gate by at most one priming window (384s) — the case where real finality has passed the denied height but the node is reading `12107` — but in this incident the two clocks expired together and the extension was ~0.

## Observed: finalized head

Healthy before the invalidation, `Verified` source live:

```
20:58:19 warn super authority finalized a block ahead of local finalized; using local finalized
chain_id=420120187 vn_id=c36d
super_authority_finalized=0xa9d4...:101636 local_finalized=0xd1c7...:101176
```

Reset creates d881, which loads the correct finalized from the EL:

```
21:00:41 info Reset of Engine is completed chain_id=420120187 vn_id=d881
local_unsafe=0x3de9...:102231 cross_unsafe=0x3de9...:102231
local_safe=0xa9d4...:101636 cross_safe=0xa9d4...:101636 finalized=0xa9d4...:101636
```

4 seconds later, published finalized is the anchor:

```
21:00:45 warn Gating unsafe ingestion during invalidation recovery
chain_id=420120187 vn_id=d881 maxDeniedHeight=102109
finalized=0x7628dfd671d42a82e505b2d323552117cef30db20faec3fadf174756010af45a:12107
```

Pinned there for d881's whole life. `Sync progress` logs `FinalizedHead()`/`SafeL2Head()` directly (`engine_controller.go:570-571`), so these are the published values:

```
21:05:22 info Sync progress chain_id=420120187 vn_id=d881 reason="consolidated block with L1"
l2_finalized=0x7628...:12107 <-- anchor, pinned, identical hash throughout
l2_safe=0x47a7...:102737 <-- safe unaffected, advancing
l2_pending_safe=...:102751 l2_unsafe=...:102751
21:05:24 info Sync progress ... l2_finalized=0x7628...:12107 l2_safe=0x10b2...:102822 l2_unsafe=...:102837
```

Recovery, after ca57 finally receives L1 finality at `21:13:14`:

```
21:16:02 warn Resuming unsafe ingestion, finality passed the invalidation
chain_id=420120187 vn_id=ca57 maxDeniedHeight=102109 finalized=0xefa3...:102109
```

**Both chains, same height, different hashes** — the tell that the value is computed from a timestamp, not read from a DB:

```
21:00:45 warn Gating ... chain_id=420120186 vn_id=b49c maxDeniedHeight=102223 finalized=0x0229...:12107
21:00:45 warn Gating ... chain_id=420120187 vn_id=d881 maxDeniedHeight=102109 finalized=0x7628...:12107
```

Earlier occurrences the same hour: `20:06:0x` and `20:52:0x`, both chains, always `...:12107`.

Anchor arithmetic (derived, self-consistent across three independent `Sync progress` lines): `l2_time - l2_unsafe.Number` = `1785271820-102128` = `1785272443-102751` = `1785271818-102126` = **1785169692**, i.e. 1s blocks and L2 genesis time `1785169692`. Block `12107` ⇒ ts `1785181799` ⇒ `activationTimestamp = 1785181800`. Exactly `activationCap()` = `activationTimestamp - 1`, and `TargetBlockNumber(1785181799) = 12107`.

## Why safe is not hit directly

The two SuperAuthority accessors differ in whether they take an L1 bound at all:

| | accessor | verifier call | with `FinalizedL1 == zero` |
|---|---|---|---|
| safe | `FullyVerifiedL2Head` | `LatestVerifiedL2Block(chainID)` — **no L1 argument** | unaffected → `Verified` → safe fine |
| finalized | `FinalizedL2Head` | `VerifiedBlockAtL1(chainID, ss.FinalizedL1)` | zero ref → `Anchor` |

Safe cannot see the zero because its query has no L1 parameter. That asymmetry is what makes the incident diagnosable, and it is why `l2_safe=102737` and `l2_finalized=12107` appear on the same log line.

## Observed: safe head, first reset — rewinds to finalized, below the last cross-safe

```
21:00:33 info Sync progress chain_id=420120187 vn_id=c36d
l2_finalized=0xa9d4...:101636 l2_safe=0xa34b...:102220 l2_unsafe=...:102230
21:00:33 info Sync progress chain_id=420120187 vn_id=c36d
l2_finalized=0xa9d4...:101636 l2_safe=0xa9d4...:101636 l2_unsafe=...:102231
^^^^^^ 102220 -> 101636 == the finalized head
21:00:41 info Resetting safe head db chain_id=420120187 vn_id=d881 l2=0xa9d4...:101636
```

Safe lands on `101636`, the **finalized** head — **584 blocks below the last cross-safe (`102220`)**, and 473 below the invalidation point (`maxDeniedHeight=102109`, so `102108` is the deepest rewind correctness requires).

Mechanism is `FindL2Heads` (`op-node/rollup/sync/start.go:282-288`): the walkback from unsafe searches for a safe candidate with a full sequence window of canonical L1 origins, and if it reaches the finalized head first it hard-assigns safe to it:

```go
// Don't traverse further than the finalized head to find a safe head
if n.Number == result.Finalized.Number {
lgr.Info("Hit finalized L2 head, returning immediately", ...)
result.Safe = n
return result, nil
}
```

```
21:00:41 info Hit finalized L2 head, returning immediately chain_id=420120187 vn_id=d881
unsafe=0x3de9...:102231 safe=0xfd6b...:102222 finalized=0xa9d4...:101636
```

So safe-rewind depth is set by **wherever the finalized head happens to be**, not by the invalidation point. Here finalized was still healthy, so the overshoot was only ~500 blocks. That coupling is what makes the anchor catastrophic on the next reset.

## Amplification: the anchor poisons the EL finalized label → 44,389-block safe rewind

`FindL2Heads` takes its bound from `currentHeads`, which reads the **EL's `finalized` label** (`start.go:78`, `L2BlockRefByLabel(eth.Finalized)`) — not `localFinalizedHead`. And the engine pushes the anchor to the EL via `fc.FinalizedBlockHash = e.FinalizedHead().Hash` (`engine_controller.go:694`, `tryUpdateEngineInternal`, on the derivation path which was live throughout — `Inserted new L2 unsafe block` 102364+ at `21:00:58`).

op-reth accepts it. Chain `420120186`, anchor block `12107` = `0x022906f9...` (hash matches the gating log above exactly):

```
21:06:47 info target=reth_node_events::node message="Forkchoice updated"
head_block_hash=0x084ebbcd... safe_block_hash=0x5fc39a4b...
finalized_block_hash=0x022906f9f37efad1a5f15fb810b85b6837fca06d3a903cddc085fa8b284f6fab
21:06:47 warn target=engine::tree forkchoice_status: Valid ... finalized_block_hash: 0x022906f9...
```

Four seconds later the next reset on chain `420120187` reads the poisoned label back out of the EL:

```
21:06:51 info Loaded current L2 heads chain_id=420120187 vn_id=ca57
unsafe=0xd3e2...:102966 safe=0x3d36...:102826
finalized=0x7628dfd671d42a82e505b2d323552117cef30db20faec3fadf174756010af45a:12107
^^^^^^^^^^^^ EL finalized label is now the anchor (was 101636)
```

With the floor at `12107` instead of `101636`, the `n.Number == result.Finalized.Number` early return never fires, so the walkback keeps descending until the sequence-window condition at `start.go:277-280` is satisfied and it exits via `if ready { result.Safe = n }` (`start.go:322`):

```
21:07:35 info Reset of Engine is completed chain_id=420120187 vn_id=ca57
local_unsafe=0xd3e2...:102966 cross_unsafe=0xd3e2...:102966
local_safe=0x7eeb...:58437 cross_safe=0x7eeb...:58437 finalized=0x7628...:12107
```

**Safe: `102826` → `58437`. A 44,389-block rewind**, requiring re-derivation of ~44k blocks.

The stopping point is quantitatively consistent with the sequence-window exit: the walkback descended `102966 → 58437` = 44,529 L2 blocks; at 1s L2 blocks and 12s L1 blocks that is ~3,711 L1 blocks of origin descent, and `SyncLookback()` returns `SeqWindowSize` (`op-node/rollup/types.go:804-811`). Wall-clock also checks out: `21:06:51 → 21:07:35` = 44s for ~44.5k single-block iterations.

Corroborating that the EL label really regressed, a later reset loads `local_finalized` far below the pre-incident value as finality re-climbs:

```
21:16:01 warn super authority finalized a block ahead of local finalized; using local finalized
chain_id=420120187 vn_id=ca57
super_authority_finalized=0xefa3...:102109 local_finalized=0x66cd...:93677
```

Full loop:

1. invalidation → vn replaced → fresh `StatusTracker`, `FinalizedL1 == zero` for up to 384s
2. `VerifiedBlockAtL1(chain, zero)` → `activationCap()` → `Anchor` → published finalized = `12107`
3. FCU sends `finalized_block_hash` = block `12107`; **op-reth returns `Valid`**; EL label regresses
4. next reset's `currentHeads` reads `12107` from the EL → walkback floor collapses → **safe rewinds ~44k blocks**
5. mass re-derivation → more resets → each new vn re-arms another 384s blind window (plausibly the engine behind #21948)

Two defects here beyond the anchor itself: the EL accepting a **backwards** `finalized_block_hash` as `Valid`, and op-node **round-tripping its own finality through the EL label** so one bad publish becomes durable input.

## Blast radius

- **44,389-block safe rewind** on the following reset, via the poisoned EL label. Worst consequence; forces mass re-derivation.
- **Engine API — backwards finalized, delivered and accepted.** op-reth was told `101636`, then `12107`, and answered `Valid`.
- **Unsafe-ingestion gate reads a meaningless input**, though it was not materially extended in this incident. `unsafeDenyGatingActive` gates on `maxDenied > FinalizedHead()`; the healthy value (`101636`) was also below `maxDenied=102109`, so the gate was inherently open until real finality passed the denied height. Worth fixing for correctness — the anchor can extend the gate by up to one priming window (384s) whenever real finality has already passed the denial — not as an availability win.
- **Every virtual node restart is exposed**, not just invalidations — any path that replaces a vn publishes the activation anchor as finalized for up to 384s. Invalidation churn is simply the most frequent trigger.
- **Dashboards/metrics** show finalized falling off a cliff on every rollback, training operators to ignore a real alarm.
- **Additional unexercised path:** `crossSafeFallback` also floors at `FinalizedHead()`, so a concurrent verifier read failure would drag cross-safe to the anchor directly, without the EL round-trip. Did not fire in this window.
- **Latent:** the same unvalidated value feeds `unsafeHead.Number < FinalizedHead().Number` → `CriticalErrorEvent` → node exit (`engine_controller.go:683`). Hard to trigger, since the Anchor branch is ceilinged at `localFinalizedHead`.

## Proposed fix

1. **Prime L1 finality at virtual-node startup.** Fetch `L1BlockRefByLabel(eth.Finalized)` once during init, or give `PollBlockChanges` an immediate first poll before entering its ticker loop (`op-service/eth/heads.go:89`). This closes the window at the source and also fixes L1 *safe* (`l1SafeSub`, same helper, same gap). Cheapest correct fix.
2. **Treat a zero `l1Block` as unknown, not as "nothing verified."** `VerifiedBlockAtL1` should return `ErrNotStarted`/hold-previous when `l1Block == (eth.L1BlockRef{})`, so the engine's existing cache path handles it instead of publishing the anchor. Note a0bbd932a7 already added an `ErrNotStarted` guard on the line *immediately above* this branch for the cold-start case — the zero-L1 case was simply not considered.
3. **Decouple the deny gate:** compare `maxDeniedHeight` against `localFinalizedHead`, which never depended on the verifier or on L1 finality signals and was correct throughout.
4. **Stop round-tripping own finality through the EL label.** Bound the `FindL2Heads` walkback on `localFinalizedHead` rather than `L2BlockRefByLabel(eth.Finalized)` (`start.go:78`), so a bad publish cannot become durable input to the next reset. This is what turns a transient 384s glitch into a 44k-block rewind.
5. **Worth raising upstream:** op-reth answered `Valid` to a forkchoice update whose `finalized_block_hash` moved *backwards* ~90k blocks. Rejecting non-monotonic finalized would contain this at the EL boundary.

(1) or (2) removes the regression; (4) is what prevents any future bad publish from causing a 44k-block rewind.

## Reproduction

Any restart or invalidation-driven replacement of a virtual node in SuperAuthority mode. `interop-reorg-*` reproduces it on every rollback — 4x in the hour examined, on all three supernodes. Tells:

- published finalized equals the same L2 height on *every* chain in the dependency set, with *different* hashes, immediately after a `vn_id` change
- no `New L1 finalized block` log for the new `vn_id` yet
- clears within one `L1EpochPollInterval` of that vn's start

Logs: `gcx logs query -d grafanacloud-logs '{namespace="an-interop-reorg-4-supernode-0"}' --from 2026-07-28T20:00:00Z --to 2026-07-28T22:00:00Z`

## Relevant code

- `op-supernode/supernode/activity/interop/interop.go` — `VerifiedBlockAtL1` (zero-`l1Block` branch), `LatestVerifiedL2Block`, `activationCap`
- `op-supernode/supernode/chain_container/super_authority.go` — `FinalizedL2Head`, `FullyVerifiedL2Head`, `verifierContribution`
- `op-node/rollup/status/status.go:132` — `OnL1Finalized`, sole writer of `FinalizedL1`
- `op-node/node/node.go:373` + `op-service/eth/heads.go:89` — un-primed ticker
- `op-node/flags/flags.go:313` — `L1EpochPollInterval` default 384s
- `op-node/rollup/driver/driver.go:55` — per-vn `StatusTracker`
- `op-node/rollup/sync/start.go` — `currentHeads:78`, finalized floor `:282-288`, sequence-window exit `:277-280`/`:322`
- `op-node/rollup/engine/engine_controller.go` — `FinalizedHead:328`, `resolveAnchorAsFinalized:378`, `crossSafeFallback:296`, `unsafeDenyGatingActive:772`, FCU `:694`, node-exit check `:683`

## Related

- #21097 — structural root cause of this bug class (SuperAuthority head accessors, anchor fallback, zero-value sentinels). This is an uncovered instance: same function, adjacent branch.
- #21092 — cold-start variant of the anchor pinning.
- #21948 — invalidation recovery repeatedly resetting; each new vn re-arms a 384s anchor window, so the two likely compound.

Contributor guide

Open the contributing guide

Research direction

Start with op-node/rollup/status/status.go, op-service/eth/heads.go, and op-supernode/supernode/chain_container/super_authority.go to trace how a new virtual node obtains FinalizedL1. Then inspect VerifiedBlockAtL1 in op-supernode/supernode/activity/interop/interop.go and the finalized forkchoice path in engine_controller.go. Done should prevent a zero L1 finality value from publishing the activation anchor or regressing the EL finalized label.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.