cockroachdb / cockroachdb/pebble

db: failing flush retries ~10^5/sec with no backoff; core burn

Open
#6,139 0 comments 0 reactions 0 assignees View on GitHub
O-community
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

hi pebblers,

appreciate all the hard work that goes into pebble and happy to chip in with this report and PR if y'all like...

## db: failing flush retries at ~10^5/sec with no backoff — a store whose directory is removed burns core until process exit

### Summary

When `flush1()` fails persistently, `DB.flush()` reports the error and then
unconditionally calls `maybeScheduleFlush()`. Because a failed flush leaves the
memtable queue exactly as it was, every precondition that triggered the flush is
still true, so a new flush goroutine starts immediately — with no delay, no
consecutive-error count, and no cap.

The result is a hot retry loop. We measured **~168k background errors in 13
seconds (~13k/sec)** from a single store, sustained until the process was
killed.

The code anticipates this exactly:

```go
if bytesFlushed, err = d.flush1(); err != nil {
// TODO(peter): count consecutive flush errors and backoff.
d.opts.EventListener.BackgroundError(err)
}
```

(`compaction.go`, `func (d *DB) flush()` — present in v2.1.6 and still on
`master` as of 2026-07-24.)

### Why it never exits

`maybeScheduleFlush()` gates on:

```go
if d.mu.compact.flushing || d.closed.Load() != nil || d.opts.ReadOnly { return }
if len(d.mu.mem.queue) <= 1 { return }
if !d.passedFlushThreshold() { return }
```

Every one of those guards asks *"is there work pending?"*. None asks *"did the
last attempt fail?"*. A failed flush is precisely the case where the work stays
pending — so **failure preserves the condition that triggers the retry**. `flush()`
sets `flushing = false` and calls `maybeScheduleFlush()` on the same path, closing
the loop with zero delay between iterations.

### Reproduction

Deterministic, no fault injection framework needed:

1. Open a DB (we hit this with `DisableWAL: true`, though the loop is not
WAL-specific).
2. Write enough to put a memtable past the flush threshold.
3. Remove the store's directory out from under the running DB
(`os.RemoveAll(dir)`).
4. Observe `EventListener.BackgroundError` firing continuously —
`creating object 084467: open /…/001/084467.sst: no such file or directory` —
and a core pinned at 100%.

Real-world trigger is not exotic: a detached volume, an evicted container mount,
an operator clearing a data directory, or any I/O error that does not heal on its
own. We reproduced it as a deterministic unit test in our own tree.

### Impact

- **CPU**: measured 28–130% of a core doing nothing but failing.
- **Log/observability**: ~13k `BackgroundError` callbacks per second. Any
listener that logs unconditionally will flood its destination; ours only
surfaced the scale because a rate-limiter reported `168927 suppressed in 13s`.
That is the dangerous part — **a well-behaved log damper turns this into a
quiet, invisible pathology** while the CPU burn continues.
- **Liveness**: the DB keeps accepting writes it can never durably persist. There
is no state a caller can observe to learn the store is unrecoverable, so the
failure is silent from the application's side.

### Suggested direction (upstream's call, obviously)

Any of these would fix the CPU burn; the last is what we'd most want as an
embedder:

1. Backoff on consecutive flush failures, as the TODO says.
2. A consecutive-failure cap after which the DB enters a terminal error state.
3. A way for the embedder to *observe* that state — an error returned from
subsequent operations, so a dead store is distinguishable from a slow one
without inferring it from callback frequency.

### Related

`Flush()` / `AsyncFlush()` on a store in this condition never complete, so any
synchronous durability call parks forever. We bound ours with a timeout, but a
terminal error state would make that unnecessary.

### Environment

- pebble `v2.1.6` (loop verified unchanged on `master`, 2026-07-24)
- Go 1.x, darwin/arm64 and linux/amd64 — reproduced on both
- `DisableWAL: true` in our configuration; the retry loop itself does not appear
WAL-dependent

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in compaction.go at func (d *DB) flush(), then trace maybeScheduleFlush() and the flush-error path. Reproduce the persistent failure by removing the store directory after crossing the flush threshold, and add a regression test that verifies retries do not continue unchecked and that the resulting failure behavior is observable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.