cockroachdb / cockroachdb/cockroach
upgrademanager: upgrade manager runs from all nodes, potentially impacting the liveness range
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The auto-upgrade process starts the upgrade from every node in the cluster. Throughout this upgrade, the upgrade manager runs scans of the entire liveness range, waiting for the cluster to be healthy and stabilize.
In a large cluster that may be experiencing intermittent problems, this can quickly result in dozens of nodes scanning the entire range in a tight (1 second backoff loop).
Typically such a read-only scan is cheap, but it has the potential unintended consequence of serialising two unrelated heartbeat updates.
It would be a substantial improvement if the upgrade manager was only running on one node.
**Short Term Ideas**
A potentially cheap (backportable) way to do this for single-tenant clusters is to use `IsMeta1Leaseholder` to decide whether the auto-upgrade loop should be running. This won't work for secondary tenants because those tenants don't have access to IsMeta1Leaseholder.
**Longer Term Idea**
The auto-upgrade process could be a job.
Jira issue: CRDB-54111
Contributor guide
Research direction
Locate the auto-upgrade loop and the existing IsMeta1Leaseholder logic, then trace how upgrade-manager scans are started on each node. Compare the short-term leaseholder approach with the longer-term job idea, including secondary tenants; done means the upgrade process runs from only one appropriate node without the current cluster-wide scan amplification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100