crypto-org-chain / crypto-org-chain/cronos-store

memiavl: WAL writer can hang Commit()/Close() indefinitely if stuck without erroring

Open
#93 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
0
Forks
3
PR merge metrics
No merged PRs in 30d

Description

## What

Follow-up to #80 (`fix/memiavl-busy-wait`). That PR bounded `waitCommittedVersion` (used by `checkBackgroundSnapshotRewrite`) to a 5s timeout instead of spinning forever. It does not cover the async WAL writer goroutine (`initAsyncCommit`, `memiavl/db.go`) hanging in other ways.

## Issue

The writer is only known to have failed when it returns an error and closes `walQuit`. If it instead blocks forever inside a syscall (unresponsive disk, NFS hang, disk full but not erroring), `walQuit` never fires, and two call sites hang with no timeout at all:

1. `Commit()` → `db.walChan <- &entry`. `walChan` is a bounded buffered channel; once the stuck writer stops draining it and it fills, every subsequent `Commit()` blocks on this send indefinitely while holding `db.mtx` — the whole state machine hangs.
2. `waitAsyncCommit()` (used by `WaitAsyncCommit()` and `Close()`) → `err := <-db.walQuit`. If the writer never exits, this blocks forever, including at node shutdown.

`checkAsyncCommit()`'s non-blocking select on `walQuit` only catches "writer errored and exited," not "writer alive but stuck."

## Solution (proposed)

Summary: track writer liveness via a progress heartbeat (atomic timestamp updated after each successful `WriteBatch`), and bound both blocking points on *idle time* since last progress rather than a fixed deadline — so a genuinely stuck writer fails fast (bounded error instead of infinite hang), while a slow-but-alive writer under a legitimate backlog isn't falsely killed.

## Test plan

- Unit test: writer that blocks forever (simulated stuck disk) with a small `walChanSize` — assert `Commit()` errors out within ~timeout instead of hanging.
- Unit test: writer that's slow but keeps advancing must NOT trigger the timeout (no false positives on legitimate backlog).
- `-race` re-run for the new atomic field.
- Manual: `Close()` during a simulated stuck writer returns promptly with an error instead of hanging shutdown.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in memiavl/db.go, tracing initAsyncCommit, Commit(), waitAsyncCommit(), WaitAsyncCommit(), Close(), and checkAsyncCommit(). Review how walChan and walQuit coordinate the writer, then run or add the proposed stuck-writer and slow-writer tests with a small walChanSize. Done means Commit() and Close() return bounded errors for a permanently blocked writer, while a progressing writer is not timed out; rerun with -race.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.