crypto-org-chain / crypto-org-chain/cronos-store
memiavl: WAL writer can hang Commit()/Close() indefinitely if stuck without erroring
- Dominant language
- Go
- Stars
- 0
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
## What
Follow-up to #80 (`fix/memiavl-busy-wait`). That PR bounded `waitCommittedVersion` (used by `checkBackgroundSnapshotRewrite`) to a 5s timeout instead of spinning forever. It does not cover the async WAL writer goroutine (`initAsyncCommit`, `memiavl/db.go`) hanging in other ways.
## Issue
The writer is only known to have failed when it returns an error and closes `walQuit`. If it instead blocks forever inside a syscall (unresponsive disk, NFS hang, disk full but not erroring), `walQuit` never fires, and two call sites hang with no timeout at all:
1. `Commit()` → `db.walChan <- &entry`. `walChan` is a bounded buffered channel; once the stuck writer stops draining it and it fills, every subsequent `Commit()` blocks on this send indefinitely while holding `db.mtx` — the whole state machine hangs.
2. `waitAsyncCommit()` (used by `WaitAsyncCommit()` and `Close()`) → `err := <-db.walQuit`. If the writer never exits, this blocks forever, including at node shutdown.
`checkAsyncCommit()`'s non-blocking select on `walQuit` only catches "writer errored and exited," not "writer alive but stuck."
## Solution (proposed)
Summary: track writer liveness via a progress heartbeat (atomic timestamp updated after each successful `WriteBatch`), and bound both blocking points on *idle time* since last progress rather than a fixed deadline — so a genuinely stuck writer fails fast (bounded error instead of infinite hang), while a slow-but-alive writer under a legitimate backlog isn't falsely killed.
## Test plan
- Unit test: writer that blocks forever (simulated stuck disk) with a small `walChanSize` — assert `Commit()` errors out within ~timeout instead of hanging.
- Unit test: writer that's slow but keeps advancing must NOT trigger the timeout (no false positives on legitimate backlog).
- `-race` re-run for the new atomic field.
- Manual: `Close()` during a simulated stuck writer returns promptly with an error instead of hanging shutdown.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in memiavl/db.go, tracing initAsyncCommit, Commit(), waitAsyncCommit(), WaitAsyncCommit(), Close(), and checkAsyncCommit(). Review how walChan and walQuit coordinate the writer, then run or add the proposed stuck-writer and slow-writer tests with a small walChanSize. Done means Commit() and Close() return bounded errors for a permanently blocked writer, while a progressing writer is not timed out; rerun with -race.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100