cockroachdb / cockroachdb/cockroach
spanconfigmanager: a newly restored or cutover tenant waits 10 minutes before reconciliation job
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Once a tenant comes online after c2c cutover or a tenant restore, its reconciliation job does not run for 10 minutes, the default value of `spanconfig.reconciliation_job.check_interval`. To understand why, consider the following timeline:
- t0: restore completes, restoring the tenant's jobs table (which includes the backed up reconciliation job)
- t1: the spanConfig manager observes a running reconciliation job in its first [run loop](https://github.com/cockroachdb/cockroach/blob/ef850fb9773921d19cfa220fd756623c4d91505c/pkg/spanconfig/spanconfigmanager/manager.go#L139), so it doesn't start a new reconciliation job
- t2: a node attempts to start the backed up reconciliation job, which [immediately succeeds](https://github.com/cockroachdb/cockroach/blob/2f811420a1492ca547659003f193233341b7fd87/pkg/spanconfig/spanconfigjob/job.go#L59) without doing any work, as the job began on a different cluster
- t_10_minutes_later: the spanConfig manager realizes no reconciliation job is running and spins up a new one
It's worth noting this bug does not affect vanilla cluster restores, as we test [here](https://github.com/cockroachdb/cockroach/blob/c3d4adeb083c4b237052533c39ace059162b71b0/pkg/ccl/backupccl/full_cluster_backup_restore_test.go#L369), because the restoring cluster begins a reconciliation job before the restore begins.
To get rid of this 10 minute wait, I propose adding a new `manager.start` method [here](https://github.com/cockroachdb/cockroach/blob/ef850fb9773921d19cfa220fd756623c4d91505c/pkg/spanconfig/spanconfigmanager/manager.go#L98) which checks and cancels any span config job started on a previous cluster.
UPDATE: after chatting with @dt, we don't think scanning the job table at tenant startup is a great idea. If there's a fix that avoids that, like applying a constant jobID to the span config job, that would be preferable.
Jira issue: CRDB-31096
Contributor guide
Assessment
This issue has not been assessed yet.