ethereum-optimism / ethereum-optimism/optimism

kona-node: unsafe block signer is log-only and has no rotation grace period (op-node reads L1 state + 3h grace)

Open
#22,839 0 comments 0 reactions 1 assignee Claimed by @joshklop View on GitHub
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

## Summary

`kona-node` and `op-node` learn the unsafe block signer from L1 in fundamentally different ways, and `kona-node` is missing the grace period that `op-node` has. During an unsafe-signer rotation on an internal devnet, every `op-node` followed the chain without interruption while every `kona-node` rejected ~25 minutes of gossip payloads, punching a hole in its unsafe chain that its EL could not heal.

All line references are against `4a44e56f7f` on `develop`.

## How op-node does it

`op-node` reads **L1 state**, not L1 events.

- [`op-node/node/runcfg/runtime_config.go:27`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L27) — the `unsafeBlockSigner` storage slot on `SystemConfig`, `keccak256("systemconfig.unsafeblocksigner")`.
- [`op-node/node/runcfg/runtime_config.go:121`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L121) — `Load()` does `ReadStorageAt(L1SystemConfigAddress, slot, l1Ref.Hash)`. It reads the absolute current value; no event history required, no dependence on having observed any particular L1 block.

It is loaded from three places:

- at startup, with a retry loop against the L1 head, before the node unblocks — [`op-node/node/node.go:382`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/node.go#L382)
- on every new L1 head
- on a poll timer — [`op-node/node/node.go:430`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/node.go#L430), flag `--l1.runtime-config-reload-interval`, default 10m ([`op-node/flags/flags.go:321`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/flags/flags.go#L321))

And it has a rotation grace period:

- [`op-node/node/runcfg/runtime_config.go:23`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L23) — `DefaultSignerGracePeriod = 3 * time.Hour`
- [`op-node/node/runcfg/runtime_config.go:139`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L139) — `rotateSigner()` keeps the previous signer alongside the new one
- [`op-node/node/runcfg/runtime_config.go:82`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L82) / [`:94`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L94) — `PreviousP2PSequencerAddress()` / `ConfirmCurrentSigner()`, consumed at [`op-node/p2p/gossip.go:478`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/p2p/gossip.go#L478) and [`:484`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/p2p/gossip.go#L484). The old signer stays valid until either the grace period expires or the first block from the new signer verifies.

The comment on that constant states the intent exactly: *"long enough to give operators time to complete a key rotation across infrastructure"*.

## How kona-node does it

`kona-node` reads **L1 logs**, and has no grace period.

- Initial value is static, from the chain registry / rollup config — [`rust/kona/crates/utilities/cli/src/flags/globals.rs:45`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/utilities/cli/src/flags/globals.rs#L45), `OPCHAINS[..].roles.unsafe_block_signer`. There is no L1 read at startup.
- Updates come only from `ConfigUpdate` logs — [`rust/kona/crates/node/service/src/actors/l1_watcher/chain.rs:81`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/service/src/actors/l1_watcher/chain.rs#L81). Note [`:91`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/service/src/actors/l1_watcher/chain.rs#L91): the filter is `.select(head_block_info.hash)` — logs of **that one head block only**.
- It is called only on live head-stream events — [`rust/kona/crates/node/service/src/actors/l1_watcher/actor.rs:153`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/service/src/actors/l1_watcher/actor.rs#L153).
- From there it flows to the gossip validator's `watch` channel — [`rust/kona/crates/node/service/src/actors/network/actor.rs:128`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/service/src/actors/network/actor.rs#L128).
- The validator is a strict equality check against the single current value — [`rust/kona/crates/node/gossip/src/block_validity.rs:103-122`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/gossip/src/block_validity.rs#L103-L122). No previous-signer concept.
- (For completeness: derivation deliberately ignores this update kind — [`rust/kona/crates/protocol/genesis/src/system/update.rs:39`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/protocol/genesis/src/system/update.rs#L39) — so the l1_watcher path above is the only one that matters for gossip.)

### Consequences of the log-based design

1. **No grace period.** The instant the `ConfigUpdate` log is seen, the old signer becomes invalid. Any skew between the L1 config update and the sequencer actually switching keys is a total gossip blackout.
2. **No bootstrap from L1 state.** A restart reverts to the static registry value, so a node that had correctly rotated is broken again by a restart.
3. **No backfill or rescan.** If the head block carrying the log is not observed live (stream hiccup, pod down, restart), the update is missed permanently.
4. **No reorg handling** for the signer value.

Issues 2-4 are latent here; issue 1 is what actually fired.

## What happened

Internal devnet, unsafe signer rotated on L1.

1. `16:45:27` — `kona-node` picks the update up promptly and correctly:
```
INFO l1_watcher Unsafe block signer update: 0x
```
The L1 watcher was at tip; nothing wrong with this part.
2. The sequencer did not begin signing with the new key until `~17:10`, ~25 minutes later.
3. For that entire window `kona-node` rejected every gossip payload. `kona_node_block_validation_failed{reason="invalid_signer"}` = **5547**, every other reason 0, nonzero only in the buckets `16:45` through `17:10`:
```
16:40 0
16:45 1078
16:50 1200
16:55 860
17:00 1200
17:05 1195
17:10 0
```
`op-node` on the same network was unaffected for the whole window — the grace period covered it exactly.
4. Result: a ~1500-block hole in kona's unsafe chain. Last insert before the blackout `number=421084` at `16:45:28`; first insert after, `number=422587` at `17:10`. kona then drove its EL forward past the hole.
5. The EL dropped into staged/pipeline sync to backfill and could not:
```
17:10:36 Preparing stage pipeline_stages=1/14 stage=Headers checkpoint=421082 target=None
17:50:38 Status connected_peers=0 stage=Headers checkpoint=421082 target=None
```
6. With the EL missing 421084, consolidation fails and the engine resets derivation back to the finalized head, forever:
```
WARN engine Received reset request err=Consolidate(MissingUnsafeL2Block(421084))
WARN engine Received `None` block for 421085
WARN engine Engine requested derivation reset
INFO derivation [SIGNAL] Reset(l2_safe_head: 420998)
WARN batch_span span batch has no new blocks after safe head
```
~122k WARN/hour. Safe head pinned, unsafe head advancing, node permanently stuck.

A second network hit the identical signer blackout and recovered, purely because its EL had L2 peers and could backfill the hole. That is luck, not design — the hole is created by kona in both cases.

### Note on observability

The rejection is metric-only — [`block_validity.rs:119`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/gossip/src/block_validity.rs#L119) increments a counter with no log line. During the blackout kona emitted no gossip or engine logs at all, which made this considerably harder to diagnose than it should have been.

## Proposed fixes

1. **Add a signer grace period to kona-node**, matching `op-node` semantics: keep the previous signer valid for a bounded period after a rotation, and clear it early once a block from the new signer verifies. Port of `runtime_config.go:23`/`:82`/`:94`/`:139` + `p2p/gossip.go:478`/`:484`.
2. **Bootstrap and periodically refresh the signer from L1 state**, i.e. `eth_getStorageAt` on `0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08` at the SystemConfig address, rather than relying solely on having observed one specific L1 head's logs. This fixes the restart, missed-head, and reorg cases together.
3. **Log at `warn!` on `invalid_signer` rejection** (rate-limited), including expected vs received. A node silently discarding all gossip should say so.

Happy to take (1) and (3).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.