ethereum-optimism / ethereum-optimism/optimism

supernode: cold-start log backfill can pin cross-safe at activation anchor during CL sync walkback

Open
#21,092 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

## Summary

Supernode cold-start log backfill currently waits for every chain to expose a stable first SafeDB entry before initializing interop verification. When a virtual node is using `consensus-layer` sync and starts with a preserved execution DB but an empty supernode/SafeDB, that chain can spend a long time inside CL sync-start walkback before its sync status and SafeDB are considered usable.

During that window, SuperAuthority has no verified head and falls back to the interop activation anchor. The engine controller can then issue FCUs with safe/finalized set to that anchor, pinning or regressing the EL safe/finalized labels even though local-safe may be advancing on one or more chains.

This appears to be a bad interaction between:

- supernode cold-start verifier/log-backfill initialization,
- all-chain SafeDB readiness gating,
- CL sync startup walkback from a preserved EL unsafe head,
- and SuperAuthority's activation-anchor fallback when verified state is empty.

## Observed behavior

Observed on a devnet supernode after wiping only the supernode DBs and preserving the reth DBs.

For chain `420120103`:

- EL latest remained high, around block `301305`.
- EL safe/finalized became block `99`, hash `0x5acda566d9e2ca7174bfe1759b19dd9db5cf4512fe35da6bd8eac20e6215bbae`.
- Supernode/op-node local-safe continued to advance well above that point, reaching around `199k`, then `212k`, then `218k` during the observed window.
- FCU finalization attempts then failed because reth still had safe at block `99`:
- `2026-05-28T15:26:56Z`: `Block must be safe before it can be finalized`, attempted finalized block `175646`, safe block `99`
- `2026-05-28T15:33:20Z`: same error, attempted finalized block `190884`, safe block `99`
- `2026-05-28T15:39:44Z`: same error, attempted finalized block `205400`, safe block `99`

The same supernode's aggregate sync status showed the pattern more clearly:

- chain `420120103`: unsafe was high, local-safe/pending-safe were high, but safe/finalized remained block `99`.
- another dependency-set chain, `420120102`, had zero L2 refs in supernode sync status.
- aggregate `safe_timestamp`, `local_safe_timestamp`, and `finalized_timestamp` were all zero.

However, direct EL RPC for chain `420120102` showed it was not empty: the preserved EL had latest around `687716` and safe/finalized around `82351`. Supernode logs showed that chain doing a very large CL sync-start walkback from the unsafe tip toward the preserved safe/finalized point.

## Relevant code paths

The code paths seem to explain the behavior:

- `op-supernode/supernode/activity/interop/interop.go`
- `tryInitFromVerifiedDB` sets `waitingForSync=true` when the verified DB is empty.
- It logs that cold start is waiting for SafeDB entries on every chain.

- `op-supernode/supernode/activity/interop/log_backfill.go`
- `advanceColdStartInit` collects every chain's first SafeDB entry timestamp.
- `collectFirstSafeHeadTimestamps` returns `ready=false` if any chain returns `ErrSafeDBNotReady`.
- Backfill and verifier initialization do not proceed until all chains are ready.

- `op-supernode/supernode/chain_container/chain_container.go`
- `FirstSafeHeadTimestamp` requires a stable first SafeDB entry.
- If SafeDB is empty, or if the deriver has not moved past the first entry's L1, it returns `ErrSafeDBNotReady`.

- `op-node/rollup/sync/start.go`
- `FindL2Heads` in CL startup walks one L2 parent at a time from the unsafe head back toward the existing safe/finalized point.
- With a preserved EL DB and a large unsafe-safe gap, this can keep the virtual node effectively uninitialized for a long time from supernode's point of view.

- `op-supernode/supernode/activity/internal/syncstatus/syncstatus.go`
- aggregate sync status uses the minimum across all chains.
- if any chain has zero timestamps / is uninitialized, the aggregate remains zero.

- `op-supernode/supernode/chain_container/super_authority.go` and `op-node/rollup/engine/engine_controller.go`
- when the verifier has no verified head, SuperAuthority returns the activation anchor.
- the engine controller resolves that anchor and uses it as safe/finalized for FCU.

## Why this looks like a bug

A supernode DB reset with preserved EL DBs should not cause a long-lived regression/pinning of EL safe/finalized to the activation anchor simply because one dependency-set chain is doing CL startup walkback.

The local-safe signal can be healthy on a chain, but cross-safe/finalized remains anchored because verifier initialization is blocked globally on another chain's SafeDB readiness. The result is confusing and potentially hazardous: the EL has a high unsafe head and may have had high safe/finalized labels before the supernode reset, but the restarted supernode can publish activation-anchor safe/finalized until all-chain cold-start conditions are satisfied.

## Suggested fixes / mitigations to consider

Possible directions:

1. Do not publish activation-anchor safe/finalized FCUs during cold start when the verified DB is empty and one or more dependency-set chains are still initializing. Prefer preserving existing EL safe/finalized until verifier initialization has completed or a newer verified head is available.
2. Make cold-start backfill able to seed from existing EL safe/finalized labels when SafeDB was reset but the EL DB was preserved, instead of requiring every chain's SafeDB to become ready first.
3. Add a bounded or explicit recovery path for `consensus-layer` sync with a large preserved unsafe-safe gap, rather than leaving interop initialization globally blocked behind parent-by-parent `FindL2Heads` walkback.
4. Add operator-facing diagnostics when cold-start backfill is blocked by a specific chain's SafeDB readiness, including the chain ID and whether CL sync-start walkback is in progress.
5. Consider rejecting or warning on the combination of empty supernode/SafeDB + preserved EL DB + CL sync + log backfill, if that combination is not intended to be supported.

A practical mitigation appears to be restarting this scenario with per-chain sync mode set to `execution-layer`, but the code should either make the CL-sync behavior safe or make the unsupported combination explicit.

Related incident thread: #21076

Contributor guide

Open the contributing guide

Research direction

Start by tracing cold-start initialization through op-supernode/supernode/activity/interop/interop.go and log_backfill.go, then follow SafeDB readiness in chain_container.go. Compare that flow with op-node/rollup/sync/start.go's FindL2Heads walkback and the activation-anchor handling in super_authority.go and engine_controller.go. Done should prevent unsafe anchor FCUs or make the unsupported preserved-EL/CL-sync case explicit, with coverage for the reported scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.