cockroachdb / cockroachdb/cockroach

backup: online restore places external files in L0 when restoring from incremental backups

Open
#170,225 3 comments 0 reactions 1 assignee Claimed by @dt View on GitHub
A-disaster-recovery A-storage C-investigation O-agent T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When restoring with EXPERIMENTAL DEFERRED COPY from a backup chain that
includes incrementals, external files from overlapping layers get placed
into L0 instead of L5. This triggers immediate compaction at ~500MB/s
that converts external files to local, defeating deferred downloads.

With a single-layer restore (AOST at the full backup timestamp), all
files land in L6 and external bytes are stable. The issue is specific to
multi-layer restores.

What we found on a 350GB TPCC fixture (tpcc-5k, 4 nodes, 8 vCPU):

The manifest dump shows the full backup file and incremental file have
identical key bounds (/Table/106/1/0 - /Table/106/2/0). The full goes to
L6, the incremental goes to L0 despite L5-L1 being empty. This is
because IngestExternalFiles (pebble) sets SkipRemoteProbe: true, which
causes pessimistic overlap detection. The exact mechanism for why it
skips past empty levels needs investigation in Pebble's ingest level
targeting code.

Reproduction:

roachprod start $CLUSTER:1-4 --args='--vmodule=replica_proposal=1'
SET CLUSTER SETTING jobs.debug.pausepoints = 'restore.before_download';
RESTORE ... AS OF SYSTEM TIME ''
WITH EXPERIMENTAL DEFERRED COPY, UNSAFE_RESTORE_INCOMPATIBLE_VERSION;
-- immediately check: curl http://localhost:26258/debug/lsm
-- observe L0 full of external vtables, L0 score > 4

Also found: the existing online restore perf benchmarks use the fixture's
FingerprintTime (which includes an incremental layer) even when
fullBackupOnly is set, so they've been measuring against a
compaction-stressed LSM rather than the intended all-L6 layout.

Areas to investigate:
- Pebble SkipRemoteProbe: why L0 when L5 is empty?
- Should restore batch overlapping files in one IngestExternalFiles call?
- Should SkipRemoteProbe be configurable per caller?

Epic: none.

Jira issue: CRDB-63892

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.