Validator epoch transition lacks stopped-check, resurrecting consensus and wiping WAL after Stop
- Dominant language
- Go
- Stars
- 22
- Forks
- 4
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 34
Description
## Details
Instance.Stop (triggered by the embedder or by context cancellation via context.AfterFunc in Start) closes stopCh and stops the running epoch. However, transitionEpochValidator (instance.go:572) - unlike transitionEpochNonValidator, which begins with an isStopped() guard (instance.go:540) - performs no stopped-check.
A validator-role epoch change is enqueued when the running epoch indexes a finalized sealing block (EpochAwareStorage.onEpochChange -> notifyEpochChange). Two interleavings lose against Stop(): (a) the change is still buffered in i.epochChanges when Stop closes stopCh, and the select in listenForEpochChanges (both cases ready) randomly picks the buffered change; or (b) listenForEpochChanges already dequeued the change and transitionEpochValidator is blocked on i.lock held by Stop - when Stop releases the lock and returns, the transition proceeds deterministically. In either case, after Stop completed: stopValidator() is a no-op (i.e already nil), i.Config.WALs is cleared, i.wal.GarbageCollect(math.MaxUint64) deletes all WAL files on disk (every WAL's retentionTerm is a round number below MaxUint64), and startAtEpoch starts a brand-new validator epoch (simplex.Epoch.Start restores from the now-empty WAL and broadcasts a replication sync) or non-validator.
Effects: (1) the instance keeps operating after Stop() returned - the new epoch may propose and sign blocks, broadcasts via the still-wired Broadcaster, and HandleMessage (which checks only i.e/i.nv for nil, never stopCh) continues dispatching peer messages into it; because Stop() early-returns once stopCh is closed, a second Stop() call cannot kill the resurrected engine. (2) All WAL files are deleted at shutdown time. The WAL is the crash-consistency mechanism preventing a validator from signing conflicting votes across restarts; deleting it while the embedder shuts down (typically to restart or hand off to a fresh Instance over the same WAL directory) removes the record of votes already cast. A restarted instance loading an empty WAL can re-vote differently in rounds the zombie already voted in, and zombie and replacement can sign concurrently with the same key - an equivocation hazard.
The window requires an epoch change in flight when Stop runs; the attacker influences only the epoch-change timing through consensus/replication traffic (e.g. timing delivery of a sealing block's finalization), not the Stop timing, so exploitability is limited; the finding is primarily a lifecycle-safety race with a security-relevant worst case.
## Evidence
1. [instance.go:572–586](https://github.com/ava-labs/Simplex/blob/main/instance.go#L572-L586)
transitionEpochValidator acquires i.lock but never checks isStopped(). After a concurrent Stop() it still wipes i.Config.WALs, garbage-collects (deletes) all WAL entries via GarbageCollect(math.MaxUint64), and calls startAtEpoch, starting a new consensus engine after the instance was stopped.
2. [instance.go:540–543](https://github.com/ava-labs/Simplex/blob/main/instance.go#L540-L543)
The sibling function transitionEpochNonValidator contains exactly the isStopped() guard that is missing in the validator path, demonstrating the intended invariant that transitions must not proceed after Stop.
3. [instance.go:237–251](https://github.com/ava-labs/Simplex/blob/main/instance.go#L237-L251)
Stop() closes stopCh and stops the running engines, then returns; nothing afterwards prevents a queued epoch change from restarting an engine. Additionally, once stopCh is closed Stop() early-returns on any later call, so the resurrected engine can never be stopped again through the public API.
4. [instance.go:351–360](https://github.com/ava-labs/Simplex/blob/main/instance.go#L351-L360)
listenForEpochChanges selects between epochChanges and stopCh; when a change is already buffered as Stop closes stopCh, Go's select picks randomly, so the stale change can be processed after Stop, invoking transitionEpochValidator. If the change was dequeued just before Stop, the transition simply blocks on i.lock and proceeds deterministically once Stop returns.
5. [instance.go:269–298](https://github.com/ava-labs/Simplex/blob/main/instance.go#L269-L298)
HandleMessage only checks i.e/i.nv for nil, not stopCh, so once the transition resurrects an engine, peer messages are again dispatched into it even though the embedder considers the instance stopped.
6. [wal/gc.go:173–191](https://github.com/ava-labs/Simplex/blob/main/wal/gc.go#L173-L191)
GarbageCollect(retentionTerm) calls Delete() on every WAL whose retentionTerm is below the argument; with math.MaxUint64 every WAL file is deleted from disk, durably destroying the vote/notarization record even if the process exits right after.
## Impact
The code establishes unauthorized continued operation after shutdown (a live signing engine the embedder believes stopped) and deletion of all WAL entries at shutdown, undermining the double-vote protection relied on across restarts (integrity LOW: limited, scenario-dependent loss; worst case equivocation requires embedder restart behavior beyond this code). Zombie goroutines/engine constitute a limited availability impact.
## Reproduction steps
1. The trigger is a coincidence of an internally queued epoch change (indirectly influenced by network peers via replication/finalization timing) with an embedder-initiated Stop or context cancellation. An attacker cannot schedule the Stop and must rely on chance plus the 50/50 select outcome, so the attack depends on runtime conditions outside attacker control. No privileges or user interaction are involved; the influencing inputs arrive over the network.
## Recommended fix
transitionEpochValidator omits the stopped-instance guard present in transitionEpochNonValidator, allowing a queued epoch change to restart consensus and destroy WAL state after Stop() completed. Fix criteria: After Stop() has run, no epoch transition may start an engine or mutate/delete WAL state; both transition paths must behave identically under a concurrent Stop. Verify by stopping the instance while a validator-role epoch change is queued and confirming no new epoch starts and WAL files are untouched.
---
**Severity:** LOW
**Status:** Open
**Category:** Race condition
**CWE:** [CWE-362](https://cwe.mitre.org/data/definitions/362.html)
**Repository:** ava-labs/Simplex
**Branch:** main
**Date created:** 2026-08-21
---
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in instance.go at transitionEpochValidator, transitionEpochNonValidator, Stop, and listenForEpochChanges to trace the concurrent shutdown path. Compare the validator transition with the existing stopped guard, then add coverage for stopping while a validator epoch change is queued. Done means no engine starts and WAL files remain untouched after Stop().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100