cockroachdb / cockroachdb/pebble
db: failing flush retries ~10^5/sec with no backoff; core burn
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
hi pebblers,
appreciate all the hard work that goes into pebble and happy to chip in with this report and PR if y'all like...
## db: failing flush retries at ~10^5/sec with no backoff — a store whose directory is removed burns core until process exit
### Summary
When `flush1()` fails persistently, `DB.flush()` reports the error and then
unconditionally calls `maybeScheduleFlush()`. Because a failed flush leaves the
memtable queue exactly as it was, every precondition that triggered the flush is
still true, so a new flush goroutine starts immediately — with no delay, no
consecutive-error count, and no cap.
The result is a hot retry loop. We measured **~168k background errors in 13
seconds (~13k/sec)** from a single store, sustained until the process was
killed.
The code anticipates this exactly:
```go
if bytesFlushed, err = d.flush1(); err != nil {
// TODO(peter): count consecutive flush errors and backoff.
d.opts.EventListener.BackgroundError(err)
}
```
(`compaction.go`, `func (d *DB) flush()` — present in v2.1.6 and still on
`master` as of 2026-07-24.)
### Why it never exits
`maybeScheduleFlush()` gates on:
```go
if d.mu.compact.flushing || d.closed.Load() != nil || d.opts.ReadOnly { return }
if len(d.mu.mem.queue) <= 1 { return }
if !d.passedFlushThreshold() { return }
```
Every one of those guards asks *"is there work pending?"*. None asks *"did the
last attempt fail?"*. A failed flush is precisely the case where the work stays
pending — so **failure preserves the condition that triggers the retry**. `flush()`
sets `flushing = false` and calls `maybeScheduleFlush()` on the same path, closing
the loop with zero delay between iterations.
### Reproduction
Deterministic, no fault injection framework needed:
1. Open a DB (we hit this with `DisableWAL: true`, though the loop is not
WAL-specific).
2. Write enough to put a memtable past the flush threshold.
3. Remove the store's directory out from under the running DB
(`os.RemoveAll(dir)`).
4. Observe `EventListener.BackgroundError` firing continuously —
`creating object 084467: open /…/001/084467.sst: no such file or directory` —
and a core pinned at 100%.
Real-world trigger is not exotic: a detached volume, an evicted container mount,
an operator clearing a data directory, or any I/O error that does not heal on its
own. We reproduced it as a deterministic unit test in our own tree.
### Impact
- **CPU**: measured 28–130% of a core doing nothing but failing.
- **Log/observability**: ~13k `BackgroundError` callbacks per second. Any
listener that logs unconditionally will flood its destination; ours only
surfaced the scale because a rate-limiter reported `168927 suppressed in 13s`.
That is the dangerous part — **a well-behaved log damper turns this into a
quiet, invisible pathology** while the CPU burn continues.
- **Liveness**: the DB keeps accepting writes it can never durably persist. There
is no state a caller can observe to learn the store is unrecoverable, so the
failure is silent from the application's side.
### Suggested direction (upstream's call, obviously)
Any of these would fix the CPU burn; the last is what we'd most want as an
embedder:
1. Backoff on consecutive flush failures, as the TODO says.
2. A consecutive-failure cap after which the DB enters a terminal error state.
3. A way for the embedder to *observe* that state — an error returned from
subsequent operations, so a dead store is distinguishable from a slow one
without inferring it from callback frequency.
### Related
`Flush()` / `AsyncFlush()` on a store in this condition never complete, so any
synchronous durability call parks forever. We bound ours with a timeout, but a
terminal error state would make that unnecessary.
### Environment
- pebble `v2.1.6` (loop verified unchanged on `master`, 2026-07-24)
- Go 1.x, darwin/arm64 and linux/amd64 — reproduced on both
- `DisableWAL: true` in our configuration; the retry loop itself does not appear
WAL-dependent
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in compaction.go at func (d *DB) flush(), then trace maybeScheduleFlush() and the flush-error path. Reproduce the persistent failure by removing the store directory after crossing the flush threshold, and add a regression test that verifies retries do not continue unchecked and that the resulting failure behavior is observable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100