kona-node: sequencer stuck in forkchoiceUpdated -38002 loop after derivation force-includes a block off its unsafe chain

A kona-node **sequencer** can be permanently wedged — hot-looping `engine_forkchoiceUpdatedV3` at thousands of calls per second until the process is unusable — whenever the engine reorgs its unsafe head while the sequencer has a block build in flight. Derivation force-inclusion after a batcher outage makes that reorg happen deterministically, so a sequencing-window expiry reliably kills the node.

Two independent defects combine:

1. `BuildTask` classifies a `-38002 Invalid forkchoice state` response as a **temporary** error, and the task queue retries temporary errors in an unbounded loop with no backoff. The condition is not transient, so the node never leaves the loop and never resets.
2. `BuildTask` has no staleness guard on the parent it was asked to build on, so it will send the EL a forkchoice state whose `safeBlockHash` is not an ancestor of `headBlockHash` in the first place.

All references below are against `68b009443f67cf830990e913bcbde8536cb15327`.

## Observed

On a devnet running a kona-node sequencer + kona-node verifier + batcher (a multi-block PoC branch, #22671 — but nothing in the mechanism is specific to it):

- The batcher submitted batches derivation could not apply.
- The sequencing window expired, so derivation force-included an empty block at height `N+1` whose parent was the last common block `N`, but whose hash differed from the sequencer's own unsafe block at `N+1`.
- The node then emitted `engine_forkchoiceUpdatedV3` → `-38002 Invalid forkchoice state` in a hot loop, thousands per second, until it was dead. It never reorged its unsafe chain onto the derived chain and never recovered.

## Mechanism

**Step 1 — derivation reorgs the unsafe chain (this part works).** The derived attributes for `N+1` fail the consolidation check against the sequencer's block at `N+1`, so [`ConsolidateTask`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/engine/src/task_queue/tasks/consolidate/task.rs#L271-L281) falls back to `build_and_seal`, and the [`InsertTask`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/engine/src/task_queue/tasks/insert/task.rs#L98-L110) that finishes it sets `unsafe_head`, `local_safe_head` and `safe_head` to the derived block. The EL and the engine state both end up on the derived chain.

**Step 2 — the sequencer submits a build against the pre-reorg head.** `SequencerActor::build_unsealed_payload` reads the unsafe head from a watch channel *once*, at [actor.rs#L217](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/service/src/actors/sequencer/actor.rs#L217), then does two pieces of network-bound async work (L1 origin selection, then `prepare_payload_attributes`, which fetches L1 receipts) before [submitting the build at L246](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/service/src/actors/sequencer/actor.rs#L245-L246). The `EngineActor` republishes the unsafe head only at the top of its step loop, [after draining the queue](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/service/src/actors/engine/actor.rs#L224-L231). So the snapshot the sequencer built against is routinely stale by the time the request lands — a window of hundreds of milliseconds, and the step-1 reorg falls squarely inside it.

**Step 3 — the FCU is rejected.** [`BuildTask::start_build`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/engine/src/task_queue/tasks/build/task.rs#L104-L110) overrides only the head with the attributes' parent and keeps the state's `safe_head`:

```rust
let new_forkchoice = state
.sync_state
.apply_update(EngineSyncStateUpdate { unsafe_head: Some(attributes_envelope.parent), ..Default::default() })
.create_forkchoice_state();
```

`head = `, `safe = `. The derived block is not an ancestor of the pre-reorg head, so the EL answers `-38002`.

**Step 4 — the hot loop.** `-38002` arrives as a JSON-RPC error and is mapped unconditionally to [`EngineBuildError::AttributesInsertionFailed`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/engine/src/task_queue/tasks/build/task.rs#L132), which carries severity [`Temporary`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/engine/src/task_queue/tasks/build/error.rs#L62-L68). `EngineTask::execute` retries temporary errors [in a `while let Err(..)` loop whose only pause is `yield_now().await`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/engine/src/task_queue/tasks/task.rs#L216-L231) — no backoff, no attempt cap. The forkchoice state is a function of engine state that nothing else can change while the queue is draining, so the retry can never succeed. That is the thousands-of-FCUs-per-second symptom.

**Step 5 — total wedge.** The loop lives inside `Engine::drain`, which the `EngineActor` runs *before* receiving its next request, so the actor can no longer process anything — including a `Reset` request. Meanwhile the sequencer is parked in `start_build_block` awaiting a payload id that will never arrive, so it cannot ask for that reset either. Nothing in the process can break the cycle.

Note the asymmetry that makes this a plain bug rather than a design trade-off: the attribute-less forkchoice update in `SynchronizeTask` **does** special-case the same error code, mapping it to [`SynchronizeTaskError::InvalidForkchoiceState`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/engine/src/task_queue/tasks/synchronize/task.rs#L125-L135) with severity `Reset`. Only the build path is missing it.

## op-node comparison

op-node guards the same two points, and this is where the fix comes from.

- **Stale parent.** [`EngineController.startBuild`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/op-node/rollup/engine/build_start.go#L57-L65) refuses a sequencer build whose parent is not the current unsafe head, before any engine call:

```go
if !attrs.IsDerived() && attrs.Parent.ID() != e.unsafeHead.ID() {
e.log.Warn("dropping stale sequencer build start", ...)
e.requestForkchoiceUpdate(ctx) // sequencer restarts on the current unsafe head
return nil, ErrStaleBuild
}
```

The `!attrs.IsDerived()` condition matters: a derivation-driven reorg legitimately builds on a parent that differs from the unsafe head.

- **`-38002` from the pre-block-creation FCU.** [`startPayload`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/op-node/rollup/engine/engine_controller.go#L1476-L1483) classifies it as `BlockInsertPrestateErr` — "pre-block-creation forkchoice update was inconsistent with engine, need reset to resolve" — which `startBuild` turns into a [`ResetEvent`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/op-node/rollup/engine/build_start.go#L104-L109), never a retry.

op-node has also already been bitten by the adjacent oscillation: see the `#21119` comment in [`FollowSource`](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/op-node/rollup/engine/engine_controller.go#L1578-L1590) — "head on the fork, safe on upstream -> InvalidForkchoiceState -> reset re-selects the fork -> oscillation" — fixed there with a `forceReset` that lands head and safe in one shot.

## Proposed fix

1. Map `-38002` in `BuildTask::start_build` to a new `EngineBuildError::InvalidForkchoiceState` with severity `Reset`, mirroring `SynchronizeTask` and op-node's `startPayload`.
2. Give `BuildTask` op-node's stale-parent guard, gated on the existing `BuildSealCoupling` flag that `SealTask` already uses for exactly this distinction (`Atomic` = derivation-driven, parent chosen from engine state, skip the check; `Detached` = external caller working from its own snapshot, check it). Report the rejection to the caller over the result channel instead of leaving it waiting.
3. Have the sequencer treat a rejected or dropped build as "re-build on the new head next tick" rather than a fatal actor error — every `NodeActor::step` error [shuts the whole node down](https://github.com/ethereum-optimism/optimism/blob/68b009443f67cf830990e913bcbde8536cb15327/rust/kona/crates/node/service/src/service/util.rs#L28-L38).
4. Publish the unsafe head after an engine reset as well, so the sequencer cannot build against a pre-reset snapshot.

Worth considering separately, not covered by the above: the task queue's temporary-error retry has no backoff at all. Any genuinely temporary engine error currently spins the engine as fast as the EL will answer. op-node's equivalent yields to a driver that re-attempts on a timer.

## Repro status

Not reproduced as an automated test. The Rust-side mechanism is covered by unit tests in the fix PR (a `-38002` build must surface as `Reset` and must not be retried; a stale detached build must be rejected without touching the engine; the sequencer must retry rather than die). A full-system repro is blocked on a harness gap:

- **No test anywhere exercises batcher outage → sequencing-window expiry → force-include → sequencer unsafe-chain reorg against a kona-node sequencer.** Not in `op-acceptance-tests/`, not in `rust/kona/tests/node/`, not in Rust.
- The one test that models the whole loop end-to-end, `TestSequencingWindowExpiry` (`op-acceptance-tests/tests/interop/seqwindow/expiry_test.go`), is bound to `presets.NewTwoL2SupernodeInterop`. Supernode presets never route through `startL2CLForKey`, so they have no `DEVSTACK_L2CL_KIND` branch and cannot start a `KonaNode`.
- Every op-e2e action test that asserts force-inclusion or post-expiry recovery (`TestL2Verifier_SequenceWindow`, `TestL2Sequencer_SequencerOnlyReorg`, `ExtendedTimeWithoutL1Batches`, `DeepReorg`) drives the in-process Go `L2Sequencer` object directly. There is no engine-API boundary to swap, so they can never target an external CL process. `rust/kona/tests/proofs/sequence_window_expiry_test.go` describes exactly this scenario but its sequencer is op-node — kona only appears there as `kona-host`.
- `rust/kona/tests/node/` has no derivation-stress package (`node/common`, `node/reorgs`, `node/restart`), and `NewMixedOpKona`/`NewMixedOpKonaForConfig` do not accept deployer options, so that preset cannot set a small `sequencerWindowSize` without a signature change.

The gap is fillable: `presets.NewMinimal(t, presets.WithDeployerOptions(sysgo.WithSequencingWindow(N)))` is accepted by the minimal preset, `startL2CLForKey` honours `DEVSTACK_L2CL_KIND=kona-node`, and the whole acceptance suite already runs against kona-node in the required `memory-all-kona-op-reth-fusaka` CI job. `dsl.L2Batcher.Stop()/Start()` and `dsl.L2CLNode.SetSequencerRecoverMode` all exist and kona implements `admin_setRecoverMode`. What is missing is the test itself. One risk worth flagging for whoever writes it: kona's derivation is gated behind EL-sync completion, which only completes after a P2P-delivered unsafe payload (see the note at `op-acceptance-tests/tests/sdm/init_test.go`).

🤖 *Co-created with Claude Fable 5*

Start with rust/kona/crates/node/engine/src/task_queue/tasks/build/task.rs and compare its error handling with synchronize/task.rs, then inspect the sequencer and engine actors cited in service/actors/. Run the relevant Rust unit tests and review op-node's build_start.go comparison. Done means stale builds are rejected safely, -38002 causes reset rather than retry, and the sequencer recovers; an end-to-end test may require the acceptance-test harness changes described.

Written by the indexing model from the issue text.