Stale queued epoch change starts a second concurrent validator epoch without stopping the first

## Details
Epoch-change notifications are queued (channel capacity 1 plus one in-flight with the listener) and processed asynchronously by listenForEpochChanges, but transitionEpochNonValidator (instance.go:536) validates them against stale assumptions.

Scenario: a node runs as a NonValidator while catching up across two epoch transitions, and is a validator at the P-chain tip (so onEpochChange notifies for every sealing block indexed, instance.go:167-168). The NonValidator indexes sealing block S1 (notification 1) and, before the listener stops it, sealing block S2 (notification 2 sits in the buffer while the listener processes notification 1).

Processing notification 1: transitionEpochNonValidator stops the NonValidator and calls startAtEpoch -> startValidator, creating and starting validator epoch E1 (which loads the configured WAL files, garbage-collects them, and begins participating - potentially proposing and signing votes).

Processing notification 2 (role still nonValidator): transitionEpochNonValidator runs again. isStopped() is false; iAmValidator(epochChange.validators) is true; stopNonValidator() is a no-op because i.nv is already nil; nothing stops the running epoch E1. startAtEpoch -> startValidator -> startEpoch then creates a second epoch E2 and overwrites i.e/i.epochOrNV (instance.go:388-389). E1 is orphaned but never stopped: its schedulers, block-building, broadcast, and signing continue.

Consequences established by the code: (a) two live consensus engines signing with the same node key; (b) because the non-validator path never clears i.Config.WALs, E2's createEpochConfig re-opens the same WAL files E1 loaded and may have deleted/GC'd and is appending to (wal.NewGarbageCollectedWAL over shared DeletableWAL handles), so the write-ahead log that exists to prevent double-voting is either shared-and-corrupted or split (E2 starts a fresh WAL unaware of votes E1 already persisted). Either way the node can emit conflicting signed proposals/votes for the same round (equivocation), a BFT safety/accountability violation, in addition to leaked goroutines and duplicated network traffic.

## Evidence
1. [instance.go:536–554](https://github.com/ava-labs/Simplex/blob/main/instance.go#L536-L554)
transitionEpochNonValidator only checks isStopped() (540) and membership in the epoch change's validator set (545). It never re-checks that the instance is still in non-validator mode: stopNonValidator() (551) is a no-op when i.nv is already nil because an earlier queued change converted the instance to a validator, and this path never calls stopValidator, yet it proceeds to startAtEpoch (553), starting a second engine while the first is live. This is the primary location the fix must change.
2. [instance.go:380–392](https://github.com/ava-labs/Simplex/blob/main/instance.go#L380-L392)
startEpoch's comment (381) states it 'assumes that the previous epoch has been stopped (if any)', but lines 388-389 unconditionally overwrite i.e and i.epochOrNV. Reached via a stale non-validator epoch change, the previously started validator epoch is orphaned but never stopped: its schedulers, timeout handler, WAL wrapper, and signing capability remain live (Epoch.Stop, which closes them, is never called).
3. [instance.go:253–267](https://github.com/ava-labs/Simplex/blob/main/instance.go#L253-L267)
stopNonValidator and stopValidator each only stop their own role's engine. transitionEpochNonValidator calls only stopNonValidator, so a running validator epoch (i.e != nil) is left untouched when the stale transition starts another one.
4. [instance.go:156–175](https://github.com/ava-labs/Simplex/blob/main/instance.go#L156-L175)
The non-validator onEpochChange closure fires once per sealing block indexed and hardcodes role nonValidator (168); it enqueues whenever the node is a validator at the P-chain tip, with no deduplication or invalidation. With the epochChanges channel of capacity 1 (instance.go:91), one notification can sit buffered while another is in flight with the listener, so two nonValidator-role changes can be outstanding: the first converts the instance to a validator, the second is processed against stale state.
5. [instance.go:429–439](https://github.com/ava-labs/Simplex/blob/main/instance.go#L429-L439)
createEpochConfig rebuilds a GarbageCollectedWAL from i.Config.WALs, which the non-validator transition path never clears. The second startValidator therefore wraps the same DeletableWAL handles the first epoch's WAL already loaded, garbage-collected (deleting files), and is appending to; maybeGarbageCollectWAL (438) can delete files the live epoch uses. If ReadAll fails on an already-deleted handle, the error propagates to processEpochChange, which halts the whole instance.
6. [instance.go:572–586](https://github.com/ava-labs/Simplex/blob/main/instance.go#L572-L586)
Only transitionEpochValidator clears i.Config.WALs (578-579). The non-validator transition path does not, establishing that the second validator start re-consumes the same WAL handles.
7. [instance.go:362–377](https://github.com/ava-labs/Simplex/blob/main/instance.go#L362-L377)
processEpochChange treats any transition error as fatal and calls i.Stop() (374-377): when the stale transition's WAL reconstruction fails because the live epoch's GC deleted shared files, the entire node halts - the availability branch of this bug.
8. [nonvalidator/non\_validator.go:234–282](https://github.com/ava-labs/Simplex/blob/main/nonvalidator/non_validator.go#L234-L282)
newFinalizedBlockTask verifies and indexes finalized blocks on the NonValidator's scheduler goroutine, holding only the NV's own lock - not the Instance lock. Each sealing block indexed triggers EpochAwareStorage.Index -> onEpochChange, producing queued notifications independently of the listener's progress; with near-adjacent sealing blocks the S2 task runs immediately after S1's, enqueuing the second notification before the listener stops the non-validator.
9. [common/sched.go:39–66](https://github.com/ava-labs/Simplex/blob/main/common/sched.go#L39-L66)
BasicScheduler.Close waits for the currently running task (running.Wait). nv.Stop -> verifier.Close therefore cannot abort an in-flight sealing-block indexing task: it completes, calls onEpochChange, and its stale nonValidator-role notification is enqueued even while the transition holding the Instance lock is stopping the NonValidator.

## Impact
If the second epoch starts, two engines sign with the same node key: the orphaned epoch's never-closed schedulers finish queued build/verify tasks and broadcast signed proposals/votes while the new epoch signs for the same rounds, and the double-sign-preventing WAL is either shared (same DeletableWAL handles re-wrapped, files closed/deleted underneath the live epoch) or split (new epoch unaware of already-persisted votes) - conflicting signed consensus messages and corrupted recovery state: integrity HIGH. If rebuilding the WAL fails because the live epoch's GC deleted shared files, processEpochChange calls i.Stop(), halting the node entirely; orphaned goroutines also leak: availability HIGH.

## Reproduction steps
1. Requires a node catching up as a NonValidator across two or more epoch transitions while being a current validator and a member of both new epochs' validator sets - a natural state for a re-joining validator. When sealing blocks are near-adjacent (low-activity chains idle across epoch boundaries), the NV scheduler indexes the second sealing block and enqueues a second nonValidator-role change before the listener stops the NonValidator, making the race nearly deterministic; otherwise remote peers widen the window by batching replication responses and flooding messages whose handling holds the Instance lock. No privileges or user interaction; the runtime-state and race preconditions make attack requirements PRESENT.

## Recommended fix
1. transitionEpochNonValidator applies a queued epoch change without re-validating that the instance is still in non-validator mode; combined with startEpoch's unconditional overwrite of i.e, a stale change starts a second engine while the first keeps running. Fix criteria: A queued epoch change must be discarded (or re-derived from current state) when the instance's role has already changed since the notification was produced; starting an epoch must be impossible while another epoch/non-validator is still running. Verify by queuing two nonValidator-role epoch changes where the first converts the node to a validator, and confirming exactly one engine instance is live afterwards.
2. The non-validator-to-validator transition path never clears i.Config.WALs, so a second startValidator reloads WAL files already owned, garbage-collected, and appended to by a live epoch, breaking the WAL's double-vote protection. Fix criteria: WAL file handles must have a single owner: once consumed by an epoch start, they must not be reloaded by a subsequent start. Verify that repeated validator starts within one Instance never construct two GarbageCollectedWALs over the same underlying files.

---
**Severity:** MEDIUM
**Status:** Open
**Category:** Race condition
**CWE:** [CWE-362](https://cwe.mitre.org/data/definitions/362.html)
**Repository:** ava-labs/Simplex
**Branch:** main
**Date created:** 2026-08-21

---

Start in instance.go at transitionEpochNonValidator, startEpoch, processEpochChange, and the role-specific stop methods; then inspect the epoch-change callback and channel setup in instance.go:156-175 and 91. Review nonvalidator/non_validator.go:234-282 and common/sched.go:39-66 to understand how queued notifications arise. Done means stale changes cannot start a second engine, only one engine owns the WALs, and the two-notification reproduction leaves exactly one live engine.

Written by the indexing model from the issue text.