cockroachdb / cockroachdb/cockroach
backup: online cluster restore of system tenant restores misses data if backup contained secondary tenants
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
An online cluster restore of the system tenant from a backup that was taken with `ElidePrefix_Tenant` (i.e. a system-tenant backup that includes virtual clusters) **silently restores empty tables** into system tenant keyspace. The link phase completes without error, but the restored tables with rewritten Ids contain no rows.
It happens to be caught for a full **cluster** restore, because the restored `system.users` is empty and the system-table restore step then panics:
```
panic: expected *DOid, found tree.dNull
tree.MustBeDOid pkg/sql/sem/tree/datum.go
roleIDSeqRestoreFunc pkg/backup/system_schema.go (SELECT max(user_id) FROM system.users -> NULL)
restoreSystemTables pkg/backup/restore_job.go
```
**To Reproduce**
1. On a **system tenant**, take a cluster backup that includes virtual clusters:
```sql
BACKUP INTO 'nodelocal://1/foo' WITH include_all_virtual_clusters;
```
2. On a fresh **system-tenant** cluster, enable copy/online restore and restore:
```sql
SET CLUSTER SETTING backup.restore.default_experimental_copy.enabled = true;
RESTORE FROM LATEST IN 'nodelocal://1/foo';
```
3. Restored tables are empty; the cluster restore panics on empty `system.users`.
**Root cause**
Elide mode is chosen at backup time (`pkg/backup/backup_job.go`):
- backup **includes tenants** (system-tenant `BACKUP ... WITH include_all_virtual_clusters`) -> `ElidePrefix_Tenant`
- backup **without** tenants (secondary tenant backing up itself) -> `ElidePrefix_TenantAndTable`
A cluster restore may reassign table IDs for every table (user tables, and system tables into a temp system DB), so under `ElidePrefix_Tenant` all those rewrites to tables with new IDs are lost. Regular (non-copy) restore is unaffected because it rewrites every key in-process before ingesting rather than relying on `SyntheticPrefix`.
Jira issue: CRDB-67856
Contributor guide
Assessment
This issue has not been assessed yet.