cockroachdb / cockroachdb/cockroach

backup: online cluster restore of system tenant restores misses data if backup contained secondary tenants

Open
#174,659 4 comments 0 reactions 1 assignee Claimed by @msbutler View on GitHub
A-disaster-recovery C-bug O-agent P-1 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

An online cluster restore of the system tenant from a backup that was taken with `ElidePrefix_Tenant` (i.e. a system-tenant backup that includes virtual clusters) **silently restores empty tables** into system tenant keyspace. The link phase completes without error, but the restored tables with rewritten Ids contain no rows.

It happens to be caught for a full **cluster** restore, because the restored `system.users` is empty and the system-table restore step then panics:

```
panic: expected *DOid, found tree.dNull
tree.MustBeDOid pkg/sql/sem/tree/datum.go
roleIDSeqRestoreFunc pkg/backup/system_schema.go (SELECT max(user_id) FROM system.users -> NULL)
restoreSystemTables pkg/backup/restore_job.go
```

**To Reproduce**

1. On a **system tenant**, take a cluster backup that includes virtual clusters:
```sql
BACKUP INTO 'nodelocal://1/foo' WITH include_all_virtual_clusters;
```
2. On a fresh **system-tenant** cluster, enable copy/online restore and restore:
```sql
SET CLUSTER SETTING backup.restore.default_experimental_copy.enabled = true;
RESTORE FROM LATEST IN 'nodelocal://1/foo';
```
3. Restored tables are empty; the cluster restore panics on empty `system.users`.

**Root cause**

Elide mode is chosen at backup time (`pkg/backup/backup_job.go`):

- backup **includes tenants** (system-tenant `BACKUP ... WITH include_all_virtual_clusters`) -> `ElidePrefix_Tenant`
- backup **without** tenants (secondary tenant backing up itself) -> `ElidePrefix_TenantAndTable`

A cluster restore may reassign table IDs for every table (user tables, and system tables into a temp system DB), so under `ElidePrefix_Tenant` all those rewrites to tables with new IDs are lost. Regular (non-copy) restore is unaffected because it rewrites every key in-process before ingesting rather than relying on `SyntheticPrefix`.

Jira issue: CRDB-67856

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.