ethereum-optimism / ethereum-optimism/optimism

kona-node: NetworkActor stalls the p2p swarm for the duration of a remote-signer HTTP call

Open
#22,646 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

## Summary

`NetworkActor::step()` awaits the block signature **inline in its `select!` arm**. With a remote signer that is an HTTP round-trip, and until it resolves `step()` has not returned — so the `select!` is not re-entered and no other arm is serviced, including the libp2p swarm event stream.

This is the kona-node counterpart of #22554, but the victim is different. In op-node a slow signer was subtracted from the next block's build window. Here the sequencer is already insulated (see below); what stalls instead is **p2p processing**.

There is also **no deadline** on the signing call, so a signer connection that hangs rather than errors wedges the network actor indefinitely.

## The code

`crates/node/service/src/actors/network/actor.rs`, one `select!` (line 107) whose `publish_rx` arm (line 136) signs before publishing:

```rust
Some(block) = self.publish_rx.recv(), if !self.publish_rx.is_closed() => {
// ...
let signature = signer.sign_block(payload_hash, chain_id, sender_address).await?; // :151 blocks here
match self.handler.gossip.publish(selector, block, signature) { // :153
```

While that future is in flight, none of the sibling arms run:

| Arm | What stops |
| --- | --- |
| `self.handler.gossip.next()` (:159) | the libp2p swarm event stream — inbound block gossip is not consumed |
| `unsafe_block_rx` | inbound unsafe blocks are not forwarded to the engine |
| `handler.enr_receiver` | discovered peers are not dialed |
| `peer_score_inspector.tick()` | peer scoring / monitoring |
| `admin_query_rx`, `p2p_rpc_rx` | admin `PostUnsafePayload`, p2p RPC |

`RemoteSigner`'s HTTP client is built without a timeout — `build_http_client` (`crates/node/sources/src/signer/remote/client.rs:103`) calls `reqwest::Client::builder()` and never sets `.timeout(...)`, and there is no timeout anywhere under `crates/node/sources/src/signer/`. reqwest's default is no request timeout.

## What is *not* affected

The sequencer does **not** block on the signer, which is worth stating because it is the part op-node got wrong:

- `SequencerActor::seal_and_commit_payload_if_applicable` hands the payload off via `schedule_execution_payload_gossip` (`actors/sequencer/actor.rs:207`).
- That trait is documented as fire-and-forget: "should return as quickly as possible and offers no guarantees that the payload actually was gossiped successfully" (`actors/network/gossip.rs`).

Two caveats on that insulation:

1. It is still a bounded `mpsc::Sender::send().await`, so it is not immune — just deeply buffered. The channel is capacity **256** (`service/node.rs:469`), i.e. ~8.5 minutes of 2s blocks before the sequencer feels backpressure.
2. 256 in-flight payload envelopes is a large memory ceiling, and the implied overflow policy is "eventually block the sequencer". For comparison, the op-node fix (#22585) deliberately chose a much smaller bound that drops the oldest entry rather than ever making the sequencer wait, on the grounds that an ungossiped block still reaches other nodes via L1 and derivation or other sync mechanisms, whereas a sequencer that stops building produces nothing for anyone. The two implementations currently disagree on that trade.

## Suggested fix

1. **Get the `await` out of the `select!` arm.** Sign and publish from a dedicated task rather than inside `step()`, so the swarm keeps being polled during the round-trip.
2. **Keep publishing ordered and bounded.** A single signing task draining a bounded queue preserves seal order — a verifier cannot follow the chain past a block it never received over gossip alone — and bounds memory well below 256 envelopes.
3. **Put a deadline on the signing call**, on the reqwest client or via `tokio::time::timeout`. This is what turns a hung signer from degraded gossip into an indefinitely wedged network actor, and it stands on its own.
4. **Choose the overflow policy deliberately**, and ideally make it agree with op-node's.

## Confidence and how to confirm

This is a **structural finding from reading the code; it has not been measured.** Whether a few hundred milliseconds of unpolled swarm meaningfully degrades inbound gossip in practice is unverified — gossipsub has internal buffering and may absorb it. The unbounded-wait case (hung signer, no timeout) is the unambiguous one.

To confirm the degraded-but-not-hung case, run a kona-node sequencer against a remote signer with injected latency and watch whether inbound unsafe-payload handling and peer scoring stall in step with the signing round-trips.

## Related

- #22554 — the op-node instance of the same shape (sequencer build window rather than p2p)
- #22585 — the op-node fix, including the ordering/bounding/overflow discussion

Contributor guide

Open the contributing guide

Research direction

Start with crates/node/service/src/actors/network/actor.rs, especially NetworkActor::step() and its publish_rx select arm, then read crates/node/sources/src/signer/remote/client.rs and related issue #22585. Trace the existing gossip and signer paths first. Done means p2p and other actor arms continue being serviced during signing, signing cannot wait indefinitely, and ordered bounded publishing behavior is deliberate.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.