cockroachdb / cockroachdb/cockroach
roachprod: add explicit public, private, and node address selectors
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Part of #77644.
Follows #173111 and #173112.
## Summary
Introduce explicit APIs for selecting a VM's public, private, or node address.
A node address is the address that the invoking roachprod process uses to reach a VM. It is selected from the cluster's persisted address mode:
- `public` mode selects the provider-reported public address;
- `private` mode selects the provider-reported private address;
- an explicit command override may select either reachable address.
Public and private selectors must remain literal. Neither may silently return the other address kind.
Design and inventory: https://github.com/cockroachlabs/cockroach/pull/2930
## Motivation
Historically, `SyncedCluster.Host()` returned `PublicIP`, and many callers used “public” or “external” to mean “reachable from the invoking process.”
#173111 allows `Host()` to fall back to `PrivateIP` when no public address exists. This enables initial private-cluster operation, but presence-based fallback is not a sufficient long-term API:
- inventory and diagnostics must distinguish real public and private addresses;
- tests may intentionally require a public endpoint;
- node access must follow the persisted mode from #173112;
- public clusters may also be reachable privately;
- callers should not independently reimplement address selection;
- transport selection, such as direct SSH versus IAP, must remain separate from address identity.
## Address kinds
### Private address
The provider-reported VPC address.
Used for node-to-node traffic and callers that explicitly request private connectivity.
### Public address
The provider-reported external address.
It may be absent. An explicit public-address request must return a clear error when no public address exists.
### Node address
The address roachprod uses to reach a VM from the invoking process.
By default:
- persisted `public` mode selects the public address;
- persisted `private` mode selects the private address.
The node address must not alter `PublicIP` or `PrivateIP`.
## Proposed API
Introduce a shared address-kind abstraction, for example:
type AddressKind string
const (
AddressKindPrivate AddressKind = "private"
AddressKindPublic AddressKind = "public"
AddressKindNode AddressKind = "node"
)
Provide one shared resolver used by `SyncedCluster` and higher-level roachprod APIs.
Possible methods include:
PrivateIP(node)
PublicIP(node)
NodeIP(node)
Address(node, kind)
The exact method names are an implementation detail, but all callers must share the same selection and error behavior.
`SyncedCluster.Host()` should delegate to the node-address selector and be deprecated or clearly documented as the node address.
## Command override
Add a command-level override for operations against existing clusters, for example:
--node-address-mode=auto|public|private
Behavior:
- `auto` uses the persisted cluster mode;
- `public` forces the public address and fails if it is absent;
- `private` forces the private address and fails if it is absent.
The override changes only how roachprod reaches an existing VM. It does not modify VM allocation, persisted cluster metadata, DNS, or advertised CockroachDB addresses.
## Transport selection
Address selection and transport selection must remain distinct.
For example, an IAP-enabled VM still has a private node address, while SSH reaches it through an IAP `ProxyCommand`.
Provide a shared node-access result or helper that combines:
- the selected node address;
- provider-specific SSH transport options.
Apply it consistently to:
- SSH sessions and `roachprod run`;
- `scp` and tree distribution;
- `rsync` and log collection;
- known-host cleanup;
- other direct roachprod-to-node commands.
## Expansion and CLI semantics
- `{ip:...}` retains its existing private-address meaning.
- `{ip:...:public}` retains literal public-address meaning and errors when absent.
- Add `{ip:...:node}` if command expansion needs the selected node address.
- `roachprod ip --external` continues to mean a literal public address.
- No command should relabel a private address as public for compatibility.
## Explicitly out of scope
- Roachtest cluster-interface migration to node endpoints.
- SQL/Admin UI URL and certificate-directory migration.
- DNS and CockroachDB advertise-address changes.
- Network-topology validation.
- Private managed instance groups and load balancers.
- Changing the default address mode to `auto`.
## Acceptance criteria
- Public, private, and node selectors are implemented through one shared resolver.
- Public and private selectors never fall back to another address kind.
- Node selection follows the persisted mode from #173112.
- The command override can force public or private access without modifying cluster metadata.
- Missing requested addresses return clear errors containing the node number or VM name.
- SSH, SCP, rsync, logs, and known-host handling use the shared node-access selection.
- IAP transport remains independent of address identity.
- Existing public clusters continue using public node addresses by default.
- Private clusters use private node addresses by default.
- Unit tests cover:
- public and private persisted modes;
- explicit overrides;
- missing public and private addresses;
- legacy inferred modes;
- IAP and direct transports;
- command expansion semantics.
Jira issue: CRDB-66425
Contributor guide
Research direction
Start with SyncedCluster address selection and the shared roachprod APIs, then trace the SSH, scp, rsync, log-collection, known-host, and command-expansion entry points. Read #173111, #173112, and the linked design inventory before choosing the resolver shape. Done means shared public, private, and node selection with overrides, clear missing-address errors, transport independence, and the listed unit-test coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100