cockroachdb / cockroachdb/cockroach
backup: online restore places external files in L0 when restoring from incremental backups
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When restoring with EXPERIMENTAL DEFERRED COPY from a backup chain that
includes incrementals, external files from overlapping layers get placed
into L0 instead of L5. This triggers immediate compaction at ~500MB/s
that converts external files to local, defeating deferred downloads.
With a single-layer restore (AOST at the full backup timestamp), all
files land in L6 and external bytes are stable. The issue is specific to
multi-layer restores.
What we found on a 350GB TPCC fixture (tpcc-5k, 4 nodes, 8 vCPU):
The manifest dump shows the full backup file and incremental file have
identical key bounds (/Table/106/1/0 - /Table/106/2/0). The full goes to
L6, the incremental goes to L0 despite L5-L1 being empty. This is
because IngestExternalFiles (pebble) sets SkipRemoteProbe: true, which
causes pessimistic overlap detection. The exact mechanism for why it
skips past empty levels needs investigation in Pebble's ingest level
targeting code.
Reproduction:
roachprod start $CLUSTER:1-4 --args='--vmodule=replica_proposal=1'
SET CLUSTER SETTING jobs.debug.pausepoints = 'restore.before_download';
RESTORE ... AS OF SYSTEM TIME ''
WITH EXPERIMENTAL DEFERRED COPY, UNSAFE_RESTORE_INCOMPATIBLE_VERSION;
-- immediately check: curl http://localhost:26258/debug/lsm
-- observe L0 full of external vtables, L0 score > 4
Also found: the existing online restore perf benchmarks use the fixture's
FingerprintTime (which includes an incremental layer) even when
fullBackupOnly is set, so they've been measuring against a
compaction-stressed LSM rather than the intended all-L6 layout.
Areas to investigate:
- Pebble SkipRemoteProbe: why L0 when L5 is empty?
- Should restore batch overlapping files in one IngestExternalFiles call?
- Should SkipRemoteProbe be configurable per caller?
Epic: none.
Jira issue: CRDB-63892
Contributor guide
Assessment
This issue has not been assessed yet.