cockroachdb / cockroachdb/cockroach

upgrademanager: upgrade manager runs from all nodes, potentially impacting the liveness range

Open
#152,916 4 comments 0 reactions 0 assignees View on GitHub
branch-master C-bug O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The auto-upgrade process starts the upgrade from every node in the cluster. Throughout this upgrade, the upgrade manager runs scans of the entire liveness range, waiting for the cluster to be healthy and stabilize.

In a large cluster that may be experiencing intermittent problems, this can quickly result in dozens of nodes scanning the entire range in a tight (1 second backoff loop).

Typically such a read-only scan is cheap, but it has the potential unintended consequence of serialising two unrelated heartbeat updates.

It would be a substantial improvement if the upgrade manager was only running on one node.

**Short Term Ideas**

A potentially cheap (backportable) way to do this for single-tenant clusters is to use `IsMeta1Leaseholder` to decide whether the auto-upgrade loop should be running. This won't work for secondary tenants because those tenants don't have access to IsMeta1Leaseholder.

**Longer Term Idea**

The auto-upgrade process could be a job.

Jira issue: CRDB-54111

Contributor guide

Open the contributing guide

Research direction

Locate the auto-upgrade loop and the existing IsMeta1Leaseholder logic, then trace how upgrade-manager scans are started on each node. Compare the short-term leaseholder approach with the longer-term job idea, including secondary tenants; done means the upgrade process runs from only one appropriate node without the current cluster-wide scan amplification.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.