cockroachdb / cockroachdb/cockroach
roachprod: persist resolved address mode in cluster metadata
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Part of #77644.
Follows #173111.
## Summary
Persist the resolved roachprod address mode so it survives `roachprod sync`, process restarts, cluster-cache regeneration, and future changes to the default GCE project.
The persisted value must be the resolved mode, `public` or `private`, rather than the original `auto` request. This prevents an existing cluster from changing behavior when provider defaults or environment configuration change.
Design and inventory: https://github.com/cockroachlabs/cockroach/pull/2930
## Motivation
#173111 introduces `--address-mode=auto|public|private`, but the resolved mode currently exists only during cluster creation. Subsequent commands infer behavior from the presence of `PublicIP`.
That inference is useful for compatibility, but it is not sufficient as the durable model:
- changing the default GCE project must not change an existing cluster;
- `sync` must recover the original resolved mode;
- future MIG grow/recreate operations must reuse the cluster's mode;
- node-address selection should not need to reinterpret provider defaults;
- inventory should distinguish intentional private allocation from malformed or incomplete VM state.
## Proposed behavior
### Creation
- Normalize and resolve the requested address mode before provisioning.
- Persist the resolved `public` or `private` value on every created VM using a stable roachprod-owned provider label, such as:
roachprod-address-mode=public
roachprod-address-mode=private
- Store the resolved mode in `vm.VM` and its serialized cluster representation.
- Never persist `auto`.
### Discovery and sync
- Read the provider label during VM discovery.
- Populate the resolved mode in `vm.VM`.
- Ensure every VM in a cluster has a compatible resolved mode.
- Report conflicting labels within one cluster as a clear metadata error.
### Backward compatibility
For existing unlabeled VMs:
- A VM with a populated `PublicIP` is inferred as `public`.
- A VM with no public IP and a populated private IP is inferred as `private`.
- A VM without either usable address is reported as malformed.
- Inference must not cause `sync` to mutate provider metadata automatically.
This preserves existing public clusters and also supports private clusters created before the durable label is introduced.
## Scope
- Add the provider-neutral resolved address-mode field to `vm.VM`.
- Add the stable provider label constant and GCE label serialization.
- Populate the field in both GCE discovery paths.
- Preserve it in the cluster cache and diagnostic output where appropriate.
- Provide shared helpers for:
- resolving a requested mode;
- inferring the mode for unlabeled VMs;
- validating consistent cluster metadata.
- Ensure ordinary lifecycle operations continue to tolerate an empty `PublicIP` in private mode.
## Explicitly out of scope
- Changing the default from `public` to `auto`.
- Introducing the general public/private/node address-selection API.
- Adding node-address overrides to operational commands.
- Private managed instance groups and load balancers.
- Implementing private allocation for non-GCE providers.
- Automatically relabeling existing clusters.
## Acceptance criteria
- Newly created GCE VMs contain the resolved address-mode label.
- `auto` is never stored as the durable value.
- `roachprod sync` reconstructs the same resolved mode after the local cache is removed.
- Changing `ROACHPROD_GCE_DEFAULT_PROJECT` does not change the mode of an existing labeled cluster.
- Unlabeled public clusters continue to resolve as public.
- Unlabeled private clusters resolve as private.
- Conflicting mode labels within one cluster produce a clear error.
- Unit tests cover:
- explicit public and private modes;
- resolved `auto`;
- labeled discovery;
- legacy inference;
- conflicting labels;
- cache serialization round trips.
Jira issue: CRDB-66424
Contributor guide
Research direction
Start with vm.VM and the roachprod creation, GCE discovery, sync, and cluster-cache paths named in the issue. Trace how address modes and provider labels move through those paths, then examine the requested unit-test coverage for resolution, legacy inference, conflicts, and cache round trips. Done means labeled clusters preserve their resolved mode while compatible unlabeled clusters still infer correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, go
- Domain
- cli, cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100