Epoch-change notification channel send under held locks can permanently deadlock the instance

## Details
Instance.notifyEpochChange (instance.go:198) does a blocking send on the capacity-1 channel i.epochChanges while its callers hold locks that the channel's only consumer needs to make progress.

For a node running in non-validator mode, every sealing block indexed fires the onEpochChange callback (instance.go:159-174) from inside NonValidator's newFinalizedBlockTask, which holds the NonValidator lock (n.lock, non_validator.go:245-263). The consumer, listenForEpochChanges, processes each notification via transitionEpochNonValidator, which needs i.lock (instance.go:537). Instance.HandleMessage acquires i.lock and then n.lock (instance.go:270,295).

This creates a lock/channel cycle: (1) a verification-task goroutine holds n.lock and blocks sending a notification because the buffer already holds an unprocessed one; (2) the listener goroutine, processing an earlier notification, blocks on i.lock; (3) a HandleMessage call holds i.lock and blocks on n.lock. None of the three can advance. A second variant needs only the listener itself: transitionEpochNonValidator, holding i.lock, calls stopNonValidator -> NonValidator.Stop -> BlockDependencyManager.Close, which waits for the running task that is blocked on the full channel.

The stopCh escape in notifyEpochChange does not help: Instance.Stop must first acquire i.lock, which is held inside the cycle, so stopCh is never closed. context.AfterFunc(ctx, i.Stop) likewise blocks. The deadlock is therefore permanent: consensus message handling, ticking, epoch transitions, and shutdown all freeze, and every subsequent HandleMessage call from the embedder blocks forever on i.lock, accumulating goroutines.

Multiple outstanding notifications are reachable: a node that is in the validator set at the P-chain tip but is catching up as a non-validator across several epoch transitions fires one notification per historical sealing block it indexes (the tip check at instance.go:167 passes each time, while transitionEpochNonValidator skips transitions for historical validator sets it is not part of, leaving the non-validator running and indexing further sealing blocks). Replication-driven indexing proceeds as fast as peers serve blocks, so consecutive sealing-block notifications can race the listener.

The validator-mode onEpochChange (instance.go:485-490) shares the same shape (notification sent from Storage.Index call paths under the engine lock and possibly i.lock), so mixed validator/non-validator transitions can also fill the buffer.

## Evidence
1. [instance.go:198–209](https://github.com/ava-labs/Simplex/blob/main/instance.go#L198-L209)
notifyEpochChange performs a blocking send on i.epochChanges. Its only escape is stopCh, but Stop() itself needs i.lock, which is unavailable in the deadlock cycle, so the escape never fires.
2. [instance.go:86–93](https://github.com/ava-labs/Simplex/blob/main/instance.go#L86-L93)
The epochChanges channel has capacity 1, so a second outstanding epoch-change notification makes the next send block.
3. [instance.go:156–175](https://github.com/ava-labs/Simplex/blob/main/instance.go#L156-L175)
The non-validator onEpochChange callback calls i.notifyEpochChange. It is invoked from EpochAwareStorage.Index, which the NonValidator calls while holding the NonValidator's lock (n.lock). Thus the blocking channel send happens under n.lock.
4. [nonvalidator/non\_validator.go:244–267](https://github.com/ava-labs/Simplex/blob/main/nonvalidator/non_validator.go#L244-L267)
newFinalizedBlockTask acquires n.lock and, still holding it, calls n.Storage.Index (line 263) which is EpochAwareStorage.Index, whose onEpochChange callback ends in the blocking channel send. The task goroutine therefore blocks on the channel while holding n.lock.
5. [instance.go:269–298](https://github.com/ava-labs/Simplex/blob/main/instance.go#L269-L298)
Instance.HandleMessage acquires i.lock and then calls i.nv.HandleMessage, which acquires n.lock. This establishes the lock ordering i.lock -> n.lock, the reverse edge of the cycle.
6. [instance.go:351–360](https://github.com/ava-labs/Simplex/blob/main/instance.go#L351-L360)
listenForEpochChanges is the only consumer of epochChanges. After receiving an item it calls processEpochChange, which blocks on i.lock inside the transition functions, so it cannot drain further notifications while i.lock is held elsewhere.
7. [instance.go:536–554](https://github.com/ava-labs/Simplex/blob/main/instance.go#L536-L554)
transitionEpochNonValidator (the consumer's processing step) acquires i.lock and calls stopNonValidator -> NonValidator.Stop -> verifier.Close, which waits for the in-flight verification task - the same task that may be blocked on the channel send, forming a second deadlock variant.
8. [common/sched.go:39–51](https://github.com/ava-labs/Simplex/blob/main/common/sched.go#L39-L51)
BasicScheduler.Close waits (running.Wait) for the currently executing task to finish, so stopping the non-validator from the listener goroutine blocks forever if that task is stuck in notifyEpochChange.

## Impact
A hit deadlock freezes the entire consensus instance permanently: no message processing, no epoch transitions, no ticking, and Stop()/context cancellation also hang, so the embedding node cannot even shut the instance down cleanly. Embedder goroutines calling HandleMessage block forever and accumulate. No confidentiality or integrity impact.

## Reproduction steps
1. Remote peers drive all inputs: replication responses pace historical block indexing (withholding then releasing a block makes queued sequences index back-to-back), and any consensus message occupies i.lock via HandleMessage. Required state: a node in the tip validator set catching up as a non-validator across several epoch transitions whose sealing blocks are closely spaced in the chain (e.g., low-activity epochs), so three notifications are in flight nearly simultaneously - Go mutex fairness bounds the listener's i.lock stall to milliseconds. The final interleaving is a race the attacker can nudge but not force; once hit, the freeze is permanent.

## Recommended fix
1. notifyEpochChange performs a blocking channel send while callers hold the NonValidator/engine locks, and the sole consumer requires i.lock (and waits for in-flight tasks) to drain the channel, creating a circular wait that also prevents Stop() from closing stopCh. Fix criteria: Epoch-change notifications must never block while any lock needed by the notification consumer (i.lock, n.lock, engine lock) is held - e.g., by making delivery non-blocking/coalescing, or by emitting notifications outside all locks. Verify by constructing the described interleaving (two queued epoch changes during non-validator catch-up plus a concurrent HandleMessage) and confirming the instance continues processing messages and can be stopped.
2. Stopping the non-validator from the epoch-change listener waits for in-flight verification tasks that can themselves be blocked publishing an epoch-change notification, a second circular wait through BasicScheduler.Close. Fix criteria: Shutting down the NonValidator/epoch from the transition path must be able to complete even when a verification task is mid-notification; verify that Stop/transition completes while a task is publishing an epoch change.

---
**Severity:** MEDIUM
**Status:** Open
**Category:** Deadlock
**CWE:** [CWE-833](https://cwe.mitre.org/data/definitions/833.html)
**Repository:** ava-labs/Simplex
**Branch:** main
**Date created:** 2026-08-21

---

Start with notifyEpochChange, listenForEpochChanges, and transitionEpochNonValidator in instance.go, then trace newFinalizedBlockTask in nonvalidator/non_validator.go and BasicScheduler.Close in common/sched.go. Reproduce the queued-notification interleaving with concurrent HandleMessage and inspect the relevant lock ordering. Done means epoch notifications cannot deadlock under held locks, transitions and Stop complete, and message processing continues.

Written by the indexing model from the issue text.