Old-epoch WAL records survive garbage collection and are replayed into the new epoch
- Dominant language
- Go
- Stars
- 22
- Forks
- 4
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 34
Description
## Details
Simplex persists consensus records (block proposals, notarizations, finalizations, empty votes/notarizations) to a WAL that is garbage-collected by round number at whole-file granularity: wal/gc.go GarbageCollect deletes only files whose highest record round is below the cutoff, and common.WALRetentionReader extracts rounds, never epochs. Rounds increase monotonically across epochs. When an epoch is sealed by a sealing block at round R (seq s), the protocol keeps producing Telocks at rounds R+1..R+k until the sealing block finalizes, and every validator writes those Telock proposals, notarizations, empty votes/notarizations, and future-seq finalizations to its WAL. On a clean transition, transitionEpochValidator wipes the WAL with GarbageCollect(MaxUint64) (instance.go:581), but that wipe runs asynchronously (channel handoff, then the listener must acquire the instance lock) after the sealing block is persisted. If the node crashes in that window, the restart path (createEpochConfig -> maybeGarbageCollectWAL) only runs GarbageCollect(R), which retains every file containing any record with round >= R, i.e. all Telock-phase records. Epoch.Start() -> restoreFromWal then replays them into the NEW epoch with no epoch filtering and no QC verification: (1) Telock blocks (seq > s, so not 'already indexed') are inserted into e.rounds; (2) setMetadataFromRecords picks the highest-round notarization/empty-notarization/finalization record - an old-epoch Telock record whose Vote.Epoch is the old epoch number - and sets e.Epoch to the OLD epoch and e.round to R+k+1, clobbering the correction epoch.Epoch = epochConfig.Epoch made in startEpoch (instance.go:387), which runs before Start(); (3) resumeFromWal rebroadcasts the stale old-epoch QC (the fresh engine's epochSealed flag is false, so broadcast is not suppressed) and, via doNotarized, signs and broadcasts a finalize vote over the old-epoch Telock header. The desynced node then rejects the new epoch's legitimate proposals (rounds R+1..R+k are occupied by Telocks in e.rounds, and header verification expects the regressed e.Epoch), while its own proposals and empty votes carry the stale epoch and are rejected by peers (msm verifyEpochNumber; simplex expected-header check), so it cannot participate in consensus until the next epoch transition rebuilds the engine. If a replayed Telock finalization at seq s+1 is later indexed, EpochAwareStorage.Index silently skips persisting it while e.lastBlock advances to the phantom block. Alternatively, if the VM rejects the Telock's nil inner-block bytes during replay, restoreFromWal errors and the node cannot start until the WAL is deleted manually. A secondary gap: transitionEpochNonValidator never clears Config.WALs, so a promoted non-validator reloads whatever stale WAL files existed at process start, guarded only by the same round-keyed heuristic. In-code TODOs (instance.go:525-527) acknowledge these scenarios are untested. In a contingent worst case where a quorum of validators crashes inside the same narrow window, they could jointly finalize an old-epoch Telock in the new epoch and stall block commitment, but that requires simultaneous similarly-timed restarts and is not attacker-inducible.
## Evidence
1. [instance.go:521–534](https://github.com/ava-labs/Simplex/blob/main/instance.go#L521-L534)
maybeGarbageCollectWAL runs on restart when the storage tip is a sealing block. It calls i.wal.GarbageCollect(md.Round), which (see wal/gc.go GarbageCollect) deletes only whole WAL files whose highest retention term (record round) is strictly below the sealing round R. Any WAL file containing old-epoch Telock records at rounds R+1..R+k (proposals, notarizations, finalizations produced after the sealing block but before its finalization, and after it while the dying epoch keeps handling messages) is retained in full and replayed into the new epoch. The TODO comments explicitly admit these scenarios are untested.
2. [simplex/epoch.go:678–726](https://github.com/ava-labs/Simplex/blob/main/simplex/epoch.go#L678-L726)
setMetadataFromRecords picks the highest-round notarization/emptyNotarization/finalization record and assigns e.round = highestRound+1 and e.Epoch = highestEpoch taken from that record. A replayed old-epoch Telock notarization at round >= e.round regresses e.Epoch to the OLD epoch number (Telocks carry the old EpochNumber) and advances e.round past the new epoch's starting round, clobbering the correct epoch number set by instance.go startEpoch.
3. [simplex/epoch.go:729–775](https://github.com/ava-labs/Simplex/blob/main/simplex/epoch.go#L729-L775)
restoreFromWal replays every record with no epoch filtering and no QC verification (WAL contents are trusted). Old-epoch Telock block records (seqs > sealing seq, so not 'already indexed') are verified with OnlyVMVerifyOpt and inserted into e.rounds; their notarizations/finalizations are attached, poisoning the new epoch's in-memory round state.
4. [simplex/epoch.go:577–665](https://github.com/ava-labs/Simplex/blob/main/simplex/epoch.go#L577-L665)
resumeFromWal then re-broadcasts the stale old-epoch notarization/finalization to the network in the new epoch and calls doNotarized(round), which signs and broadcasts a finalize vote for the old-epoch Telock and starts the (wrong) round. metadata() will subsequently build new proposals with prev = Telock digest, seq past the phantom block, and the regressed Epoch number.
5. [instance.go:382–391](https://github.com/ava-labs/Simplex/blob/main/instance.go#L382-L391)
startEpoch corrects epoch.Epoch = epochConfig.Epoch (the new epoch number) BEFORE calling epoch.Start(). Because setMetadataFromRecords runs inside Start()->restoreFromWal, the replayed stale records overwrite this correction.
6. [instance.go:536–553](https://github.com/ava-labs/Simplex/blob/main/instance.go#L536-L553)
Secondary path: transitionEpochNonValidator never clears i.Config.WALs (unlike transitionEpochValidator at line 579), so a non-validator that becomes a validator reloads whatever stale WAL files were present at process start; only the retention-term heuristic in maybeGarbageCollectWAL stands between those records and the new epoch.
7. [wal/gc.go:173–190](https://github.com/ava-labs/Simplex/blob/main/wal/gc.go#L173-L190)
GarbageCollect works at whole-file granularity keyed on the file's highest retention term (round). A file mixing records below and at/above the sealing round is kept entirely; records are never filtered by epoch.
## Impact
The restarted validator is desynchronized for the remainder of the epoch: e.Epoch regresses to the sealed epoch and e.round jumps ahead, so peers reject all its proposals and votes and it rejects theirs; it also re-signs and broadcasts finalize/empty votes for old-epoch blocks, and e.lastBlock can advance to a Telock never persisted (phantom parent for its proposals). Persisted storage stays correct (epoch check in Index) and the node still replicates finalized blocks, so integrity impact is LOW. Its consensus participation is lost until the next epoch transition or manual WAL deletion - and in the alternate branch it fails to start at all - hence availability HIGH.
## Reproduction steps
1. No attacker action triggers this: the poisoning records are legitimate protocol messages every validator writes to its WAL during each epoch transition's Telock phase. The trigger is a crash/kill/power loss landing between the sealing block's storage Index and the asynchronous WAL wipe in transitionEpochValidator - a narrow, purely operational timing window. Exploitation is bound to local crash-recovery, not to any network-deliverable input; no privileges or user interaction are involved. AT is PRESENT for the restrictive runtime condition: an epoch transition in progress, a crash inside the window, and the node a validator in the new epoch.
## Recommended fix
1. WAL garbage collection is keyed on round numbers at whole-file granularity, but rounds are shared across epochs; records from a sealed epoch with rounds >= the sealing round survive into the next epoch's WAL. Fix criteria: After a restart where the storage tip is a sealing block, no record produced in the sealed epoch (identified by its epoch number, not its round) may be replayed into the new epoch. Verify by writing Telock proposals/notarizations/finalizations to the WAL, restarting after the sealing block is indexed but before WAL wipe, and asserting the new epoch starts at round R+1 with the correct epoch number and an empty rounds map.
2. restoreFromWal and setMetadataFromRecords trust every WAL record without checking that its epoch matches the epoch being started, allowing e.Epoch to be regressed and e.rounds to be populated with foreign-epoch blocks. Fix criteria: WAL replay must ignore (or fail loudly on) records whose epoch differs from the epoch under restoration, and setMetadataFromRecords must never move e.Epoch backwards from the value derived from storage/config. Verify with a WAL containing mixed-epoch records: replay must not change e.Epoch and must not load old-epoch rounds.
3. transitionEpochNonValidator does not clear Config.WALs, so stale WAL handles from process start are reloaded when a non-validator is promoted to validator. Fix criteria: Promotion from non-validator to validator must start from a WAL state that provably contains no records from epochs before the current one. Verify by promoting a node whose Config.WALs holds old-epoch records and asserting none are replayed.
---
**Severity:** MEDIUM
**Status:** Open
**Category:** Incomplete cleanup
**CWE:** [CWE-459](https://cwe.mitre.org/data/definitions/459.html)
**Repository:** ava-labs/Simplex
**Branch:** main
**Date created:** 2026-08-21
---
Contributor guide
No contributing guide indexed for this repository
Research direction
Read wal/gc.go GarbageCollect and common.WALRetentionReader first, then trace instance.go maybeGarbageCollectWAL, startEpoch, and transitionEpochNonValidator into simplex/epoch.go restoreFromWal, setMetadataFromRecords, and resumeFromWal. Reproduce the mixed-epoch WAL restart described in the issue. Done means old-epoch records are not replayed, the new epoch and starting round remain correct, and promotion does not reload stale WAL state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100