ethereum-optimism / ethereum-optimism/optimism
op-supernode: live logsDB parent mismatch after an L2 reorg causes infinite verification retry
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 145
Description
## Summary
When a canonical L2 reorg replaces a block at or before the latest block already sealed in op-supernode's logsDB, the live interop verifier can become permanently stuck.
The execution layer may successfully adopt and serve the replacement canonical branch. However, the interop activity's next `DecisionAdvance` attempts to seal a block whose parent is on the replacement branch, while logsDB still ends on the old branch. `persistFrontierLogs` correctly rejects the append with `ErrParentHashMismatch`, but the live loop does not schedule logsDB/accepted-state reconciliation. It retries the same impossible append indefinitely.
This is the online/main-loop counterpart to #20627, which covered reorg divergence discovered during restart/backfill.
## Observed behavior
Generic error pattern:
```text
failed to progress and record interop
activity=interop
err="persist frontier logs: chain : block parent hash
does not match logsDB last sealed block hash :
block parent hash does not match logsDB"
```
At the same time, the engine-controller/super-authority path reports the same branch split:
```text
super authority safe head non-canonical (reorg signal)
chain_id=
super_authority_safe=:N
canonical=:N
```
The parent mismatch repeats on every verifier backoff. No logsDB rewind, clear, or accepted-state rewind transition is created.
## Important distinction
The L2 execution-chain reorg can appear successful:
- the canonical EL imports the replacement blocks;
- public RPC serves the replacement branch;
- transactions from the invalid branch disappear.
But op-supernode remains internally inconsistent:
```text
canonical EL: replacement block N
logsDB tip: old block N
super-authority safe: old block N
next verified block: replacement N+1, whose parent is replacement N
```
Thus the externally visible chain is repaired while interop verification is unable to advance.
## Expected behavior
The logsDB append guard should continue rejecting a block whose parent does not match the sealed tip. That invariant is correct.
After detecting that the sealed tip is no longer canonical, op-supernode should enter a durable reconciliation flow that:
1. finds the latest logsDB/verifiedDB block still canonical in the EL;
2. persists a rewind/reconciliation transition;
3. reconciles verifiedDB, logsDB, and accepted super-authority state to that boundary;
4. applies any required deny-list/engine coordination;
5. resumes verification on the replacement branch;
6. remains idempotent across crash/restart.
## Current behavior and likely gap
`DecisionAdvance` calls `persistFrontierLogs`. In `sealBlockDataIntoLogsDB`, the implementation reads `LatestSealedBlock()` and returns `ErrParentHashMismatch` when the fetched canonical block's parent differs from the logsDB tip.
The error aborts the advance before committing the verified result. The next round then retries the same advance against unchanged persisted state.
Repair mechanisms already exist:
- `applyRewindPlan` can rewind logsDB to a previous verified head when a persisted `DecisionRewind` is built and applied.
- `reconcileLogsDBTail` can trim a divergent logsDB tail during startup/backfill.
- raft-wal logsDB supports `Rewind` and `Clear`.
The missing connection appears to be from a live `ErrParentHashMismatch` / non-canonical sealed tip to one of these durable repair paths.
A local mutation of logsDB inside `DecisionAdvance` is probably insufficient because logsDB must remain coordinated with verifiedDB, pending-transition WAL, super-authority state, deny lists, and potentially chain engines.
## Reproduction outline
1. Run op-supernode with interop activity enabled and allow logsDB/verifiedDB to seal through block `N`.
2. Produce an invalid executing-message block at `N+1` and allow the unsafe branch to advance.
3. Trigger recovery that rebuilds from `N` or earlier, so canonical block `N` changes hash rather than preserving the original valid parent.
4. Allow the EL to import replacement blocks `N` and `N+1`.
5. Observe that canonical `N+1` has replacement `N` as its parent, while logsDB still ends on old `N`.
6. Observe repeated `ErrParentHashMismatch` from `persistFrontierLogs` and no verification progress.
A normal invalidation that preserves block `N` and replaces only `N+1` does not trigger this condition because the replacement block still extends the sealed logsDB tip.
## Impact
- Interop verification stalls permanently without operator intervention.
- Super-authority safe state remains on a non-canonical block.
- Kubernetes process/readiness may remain healthy despite no interop progress.
- Public RPC can look healthy, hiding the failed interop state convergence.
- Repeated errors create high log volume.
- Consensus-aware downstream routing may consume stale or regressed sync status.
- The failure has been observed more than once under recovery paths that replace an already-sealed parent, so it is not specific to concurrent invalid messages.
## Suggested direction
When `persistFrontierLogs` returns `ErrParentHashMismatch` or `ErrStaleLogsDB`:
1. Confirm the stored tip is non-canonical at its height.
2. Locate the latest common canonical sealed/verified ancestor.
3. Build and persist a `DecisionRewind`-style reconciliation plan instead of retrying `DecisionAdvance`.
4. Apply reconciliation atomically/idempotently across verifiedDB and logsDB.
5. Resume verification from the reconciled boundary.
6. Fail readiness or expose a dedicated degraded metric while this mismatch is unresolved.
7. Rate-limit repeated mismatch logs.
## Acceptance criteria
- [ ] Live same-height replacement at the logsDB tip self-recovers without restart or manual DB deletion.
- [ ] Replacement below the logsDB tip rewinds to the latest common canonical ancestor.
- [ ] verifiedDB, logsDB, super-authority safe state, and canonical EL converge on the replacement branch.
- [ ] Recovery is idempotent across crashes before and after logsDB rewind.
- [ ] A pending reconciliation transition survives restart and completes.
- [ ] Multi-chain interop state remains consistent during partial failure/retry.
- [ ] Finalized accepted history is never rewound incorrectly.
- [ ] Successful invalidation preserving the valid parent continues to use the normal fast path.
- [ ] Readiness/metrics expose a persistent unresolved logsDB/canonical mismatch.
## Relevant code
- `op-supernode/supernode/activity/interop/logdb.go`
- `sealBlockDataIntoLogsDB`
- `ErrParentHashMismatch`
- `ErrStaleLogsDB`
- `op-supernode/supernode/activity/interop/interop.go`
- `applyPendingTransition`
- `DecisionAdvance`
- `applyRewindPlan`
- `op-supernode/supernode/activity/interop/log_backfill.go`
- `reconcileLogsDBTail`
- `op-supernode/supernode/activity/interop/raftwallogdb/db.go`
- `LatestSealedBlock`
- `Rewind`
- `Clear`
## Related
- #20627 — handles divergent logsDB state discovered while the supernode is offline/restarting; this issue covers divergence encountered while the live verifier is already running.
Contributor guide
Research direction
Start in op-supernode/supernode/activity/interop/interop.go at DecisionAdvance and applyPendingTransition, then trace persistFrontierLogs and sealBlockDataIntoLogsDB in logdb.go alongside applyRewindPlan. Compare the existing reconciliation path in log_backfill.go and raftwallogdb/db.go; done means live parent mismatches durably reconcile all listed state, survive restart, and preserve the normal fast path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- blockchain, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100