cockroachdb / cockroachdb/cockroach

spanconfigmanager: a newly restored or cutover tenant waits 10 minutes before reconciliation job

Open
#109,771 6 comments 0 reactions 0 assignees View on GitHub
C-bug P-3 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Once a tenant comes online after c2c cutover or a tenant restore, its reconciliation job does not run for 10 minutes, the default value of `spanconfig.reconciliation_job.check_interval`. To understand why, consider the following timeline:

- t0: restore completes, restoring the tenant's jobs table (which includes the backed up reconciliation job)
- t1: the spanConfig manager observes a running reconciliation job in its first [run loop](https://github.com/cockroachdb/cockroach/blob/ef850fb9773921d19cfa220fd756623c4d91505c/pkg/spanconfig/spanconfigmanager/manager.go#L139), so it doesn't start a new reconciliation job
- t2: a node attempts to start the backed up reconciliation job, which [immediately succeeds](https://github.com/cockroachdb/cockroach/blob/2f811420a1492ca547659003f193233341b7fd87/pkg/spanconfig/spanconfigjob/job.go#L59) without doing any work, as the job began on a different cluster
- t_10_minutes_later: the spanConfig manager realizes no reconciliation job is running and spins up a new one

It's worth noting this bug does not affect vanilla cluster restores, as we test [here](https://github.com/cockroachdb/cockroach/blob/c3d4adeb083c4b237052533c39ace059162b71b0/pkg/ccl/backupccl/full_cluster_backup_restore_test.go#L369), because the restoring cluster begins a reconciliation job before the restore begins.

To get rid of this 10 minute wait, I propose adding a new `manager.start` method [here](https://github.com/cockroachdb/cockroach/blob/ef850fb9773921d19cfa220fd756623c4d91505c/pkg/spanconfig/spanconfigmanager/manager.go#L98) which checks and cancels any span config job started on a previous cluster.

UPDATE: after chatting with @dt, we don't think scanning the job table at tenant startup is a great idea. If there's a fix that avoids that, like applying a constant jobID to the span config job, that would be preferable.

Jira issue: CRDB-31096

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.