ethereum-optimism / ethereum-optimism/optimism
kona-node: unsafe block signer is log-only and has no rotation grace period (op-node reads L1 state + 3h grace)
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 145
Description
## Summary
`kona-node` and `op-node` learn the unsafe block signer from L1 in fundamentally different ways, and `kona-node` is missing the grace period that `op-node` has. During an unsafe-signer rotation on an internal devnet, every `op-node` followed the chain without interruption while every `kona-node` rejected ~25 minutes of gossip payloads, punching a hole in its unsafe chain that its EL could not heal.
All line references are against `4a44e56f7f` on `develop`.
## How op-node does it
`op-node` reads **L1 state**, not L1 events.
- [`op-node/node/runcfg/runtime_config.go:27`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L27) — the `unsafeBlockSigner` storage slot on `SystemConfig`, `keccak256("systemconfig.unsafeblocksigner")`.
- [`op-node/node/runcfg/runtime_config.go:121`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L121) — `Load()` does `ReadStorageAt(L1SystemConfigAddress, slot, l1Ref.Hash)`. It reads the absolute current value; no event history required, no dependence on having observed any particular L1 block.
It is loaded from three places:
- at startup, with a retry loop against the L1 head, before the node unblocks — [`op-node/node/node.go:382`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/node.go#L382)
- on every new L1 head
- on a poll timer — [`op-node/node/node.go:430`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/node.go#L430), flag `--l1.runtime-config-reload-interval`, default 10m ([`op-node/flags/flags.go:321`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/flags/flags.go#L321))
And it has a rotation grace period:
- [`op-node/node/runcfg/runtime_config.go:23`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L23) — `DefaultSignerGracePeriod = 3 * time.Hour`
- [`op-node/node/runcfg/runtime_config.go:139`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L139) — `rotateSigner()` keeps the previous signer alongside the new one
- [`op-node/node/runcfg/runtime_config.go:82`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L82) / [`:94`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/node/runcfg/runtime_config.go#L94) — `PreviousP2PSequencerAddress()` / `ConfirmCurrentSigner()`, consumed at [`op-node/p2p/gossip.go:478`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/p2p/gossip.go#L478) and [`:484`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/op-node/p2p/gossip.go#L484). The old signer stays valid until either the grace period expires or the first block from the new signer verifies.
The comment on that constant states the intent exactly: *"long enough to give operators time to complete a key rotation across infrastructure"*.
## How kona-node does it
`kona-node` reads **L1 logs**, and has no grace period.
- Initial value is static, from the chain registry / rollup config — [`rust/kona/crates/utilities/cli/src/flags/globals.rs:45`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/utilities/cli/src/flags/globals.rs#L45), `OPCHAINS[..].roles.unsafe_block_signer`. There is no L1 read at startup.
- Updates come only from `ConfigUpdate` logs — [`rust/kona/crates/node/service/src/actors/l1_watcher/chain.rs:81`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/service/src/actors/l1_watcher/chain.rs#L81). Note [`:91`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/service/src/actors/l1_watcher/chain.rs#L91): the filter is `.select(head_block_info.hash)` — logs of **that one head block only**.
- It is called only on live head-stream events — [`rust/kona/crates/node/service/src/actors/l1_watcher/actor.rs:153`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/service/src/actors/l1_watcher/actor.rs#L153).
- From there it flows to the gossip validator's `watch` channel — [`rust/kona/crates/node/service/src/actors/network/actor.rs:128`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/service/src/actors/network/actor.rs#L128).
- The validator is a strict equality check against the single current value — [`rust/kona/crates/node/gossip/src/block_validity.rs:103-122`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/gossip/src/block_validity.rs#L103-L122). No previous-signer concept.
- (For completeness: derivation deliberately ignores this update kind — [`rust/kona/crates/protocol/genesis/src/system/update.rs:39`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/protocol/genesis/src/system/update.rs#L39) — so the l1_watcher path above is the only one that matters for gossip.)
### Consequences of the log-based design
1. **No grace period.** The instant the `ConfigUpdate` log is seen, the old signer becomes invalid. Any skew between the L1 config update and the sequencer actually switching keys is a total gossip blackout.
2. **No bootstrap from L1 state.** A restart reverts to the static registry value, so a node that had correctly rotated is broken again by a restart.
3. **No backfill or rescan.** If the head block carrying the log is not observed live (stream hiccup, pod down, restart), the update is missed permanently.
4. **No reorg handling** for the signer value.
Issues 2-4 are latent here; issue 1 is what actually fired.
## What happened
Internal devnet, unsafe signer rotated on L1.
1. `16:45:27` — `kona-node` picks the update up promptly and correctly:
```
INFO l1_watcher Unsafe block signer update: 0x
```
The L1 watcher was at tip; nothing wrong with this part.
2. The sequencer did not begin signing with the new key until `~17:10`, ~25 minutes later.
3. For that entire window `kona-node` rejected every gossip payload. `kona_node_block_validation_failed{reason="invalid_signer"}` = **5547**, every other reason 0, nonzero only in the buckets `16:45` through `17:10`:
```
16:40 0
16:45 1078
16:50 1200
16:55 860
17:00 1200
17:05 1195
17:10 0
```
`op-node` on the same network was unaffected for the whole window — the grace period covered it exactly.
4. Result: a ~1500-block hole in kona's unsafe chain. Last insert before the blackout `number=421084` at `16:45:28`; first insert after, `number=422587` at `17:10`. kona then drove its EL forward past the hole.
5. The EL dropped into staged/pipeline sync to backfill and could not:
```
17:10:36 Preparing stage pipeline_stages=1/14 stage=Headers checkpoint=421082 target=None
17:50:38 Status connected_peers=0 stage=Headers checkpoint=421082 target=None
```
6. With the EL missing 421084, consolidation fails and the engine resets derivation back to the finalized head, forever:
```
WARN engine Received reset request err=Consolidate(MissingUnsafeL2Block(421084))
WARN engine Received `None` block for 421085
WARN engine Engine requested derivation reset
INFO derivation [SIGNAL] Reset(l2_safe_head: 420998)
WARN batch_span span batch has no new blocks after safe head
```
~122k WARN/hour. Safe head pinned, unsafe head advancing, node permanently stuck.
A second network hit the identical signer blackout and recovered, purely because its EL had L2 peers and could backfill the hole. That is luck, not design — the hole is created by kona in both cases.
### Note on observability
The rejection is metric-only — [`block_validity.rs:119`](https://github.com/ethereum-optimism/optimism/blob/4a44e56f7f/rust/kona/crates/node/gossip/src/block_validity.rs#L119) increments a counter with no log line. During the blackout kona emitted no gossip or engine logs at all, which made this considerably harder to diagnose than it should have been.
## Proposed fixes
1. **Add a signer grace period to kona-node**, matching `op-node` semantics: keep the previous signer valid for a bounded period after a rotation, and clear it early once a block from the new signer verifies. Port of `runtime_config.go:23`/`:82`/`:94`/`:139` + `p2p/gossip.go:478`/`:484`.
2. **Bootstrap and periodically refresh the signer from L1 state**, i.e. `eth_getStorageAt` on `0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08` at the SystemConfig address, rather than relying solely on having observed one specific L1 head's logs. This fixes the restart, missed-head, and reorg cases together.
3. **Log at `warn!` on `invalid_signer` rejection** (rate-limited), including expected vs received. A node silently discarding all gossip should say so.
Happy to take (1) and (3).
Contributor guide
Assessment
This issue has not been assessed yet.