cockroachdb / cockroachdb/pebble

db: ingestTargetLevel places external files in L0 when L1..baseLevel-1 are empty

Open
#6,074 7 comments 0 reactions 0 assignees View on GitHub
A-write-amp C-bug C-performance
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

**Summary**

`ingestTargetLevel`'s level-search loop starts at `baseLevel`, not `L1`. The function initializes `targetLevel = 0` and walks `for level := baseLevel; level < numLevels; level++`, updating `targetLevel` to the deepest empty level it visits before returning when it hits data overlap.

When the very first iteration (at `baseLevel`) sees data overlap, the function returns the un-updated `targetLevel = 0` — placing the file in L0 even though L1..baseLevel-1 are physically empty. Per Pebble's own compaction-invariant model those levels are "not opened yet" because the LSM is small, but they're still legal placement targets and would dramatically reduce L0 stacking when the alternative is the fallthrough to L0.

This shows up in the wild when an LSM has substantial data at L6 but little above (so `baseLevel` is high) and ingests arrive with overlapping bounds — each subsequent overlapping ingest collapses to L0 instead of cascading into the empty intermediate levels. The symptom is documented from the CockroachDB side at cockroachdb/cockroach#170225 (online restore links a chain of overlapping external SSTs and tens of GB pile up in L0).

**Standalone reproduction**

A small standalone Go program reproduces this in ~2 seconds:

1. Open a pebble DB with `RemoteStorage` configured (in-memory `remote.NewInMem()` is enough) and `DisableAutomaticCompactions: true` so placement decisions are observable.
2. Build three external sstables in the remote store. Disjoint point keys, but each declared with identical user-key bounds `[a, m)`. Set `Size: 128 << 20` (realistic backup-style file size) so `baseLevel` doesn't collapse to L6.
3. Ingest each with a separate `IngestExternalFiles` call.

Observed placement:

```
ingest #1 → L6 (empty LSM)
ingest #2 → L5 (overlap at L6, baseLevel is high, cascade reaches L5)
ingest #3 → L0 (overlap at L5 — loop returns initial targetLevel=0)
```

Expected (and what would happen if the loop started at L1):

```
ingest #1 → L6
ingest #2 → L5
ingest #3 → L4
```

After warming the LSM with ~940 MiB of non-overlapping data to push `baseLevel` down to L4, ingest #3 cascades to L4 as expected. So the trigger is unambiguously "loop starts at `baseLevel`."

**Code reference**

`ingestTargetLevel` in `ingest.go`. The loop initialization is `for level := baseLevel; level < numLevels; level++`. Changing the lower bound from `baseLevel` to `1` (keeping the `lsmOverlap[0]` fast-path check above the loop) lets the loop consider empty intermediate levels. A branch at https://github.com/dt/pebble/tree/ingest-cascade-from-l1 makes that one-line change with a regression-test fixture; the integration test on the CockroachDB side that previously showed 3.3 GB ingested at L0 dropped to 0 with that fix applied.

**Open questions**

- Is `baseLevel` the right floor for the search at all? The current comment block on the function explains the size-ratio reasoning, but that reasoning is about *steady-state compaction shape*, not about whether a given level is a legal ingest target. If placement above `baseLevel` is legal (it is — nothing else in pebble seems to assume otherwise), the search should consider those levels and let the compaction picker reconcile the size invariant over time.
- The compaction picker may target a file placed above `baseLevel` for an aggressive level-compaction (since its target_bytes ≈ 0 above `baseLevel`), potentially defeating the placement by triggering a materializing merge. Worth thinking about whether a compaction-side change is needed in tandem — or whether the picker's natural behavior (trivial-move-down when destination is empty) is enough.

**Workaround**

CockroachDB can mitigate symptom on its side, but the placement decision is upstream of any caller and any code path that does many overlapping external ingests into a small LSM would hit this.

Related: cockroachdb/cockroach#170225.

Jira issue: PEBBLE-1445

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.