cockroachdb / cockroachdb/cockroach
roachprod: make startup, certificates, and DNS address-mode aware
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Part of #77644.
Follows #173112, #173113, and #173114.
## Summary
Make CockroachDB startup, secure certificates, URL generation, and DNS lifecycle use the cluster's resolved address mode and explicit public/private/node address selectors.
Private clusters must start and remain operable without public IPs or public DNS records. Public-address behavior must remain available for public clusters and tests that deliberately require it.
Design and inventory: https://github.com/cockroachlabs/cockroach/pull/2930
## Motivation
Several roachprod paths still assume that every VM has a public address:
- multi-VPC clusters advertise `Host()`, historically the public IP;
- certificate generation includes `Host()` and load-balancer addresses without consistently checking whether they are present;
- SQL and Admin UI URL generation prefers public DNS and falls back to `PublicIP`;
- service-discovery records use `PublicDNS` as node identity;
- DNS cleanup assumes the current cluster still has public endpoints.
These assumptions can produce empty URLs or certificate SANs, advertise unreachable private addresses across unrouted networks, or publish private addresses into public DNS.
## CockroachDB advertise and join addresses
Node-to-node address selection must depend on network topology and the resolved cluster mode.
### Same routable network
When nodes share a canonical VPC/network identity:
- use private addresses for `--advertise-addr`;
- use private addresses for join targets;
- continue to do so for both public and private allocation modes.
Ordinary multi-region GCE clusters in one VPC fall into this category.
### Different networks
For public-mode clusters spanning different VPCs or providers:
- preserve the existing public advertise/join behavior.
For private-mode clusters spanning different canonical networks:
- require an explicit declaration that the private networks are routed;
- use private addresses when that declaration is present;
- otherwise fail before starting CockroachDB with a clear topology error.
The exact representation of routed-network reachability may be provider metadata or an explicit cluster/start option, but it must not be inferred merely from the presence of private IPs.
## Node-facing URLs
Add node-address support to roachprod URL generation:
- SQL URLs;
- Admin UI URLs;
- other generic node service URLs.
Requirements:
- node URLs use the node selector from #173113;
- explicit public URL requests retain literal public semantics;
- explicit private URL requests retain literal private semantics;
- private clusters do not fall back to an empty public DNS name or public IP;
- certificate-directory selection remains independent of address selection.
For example, a secure node URL used by a process on the invoking machine may use a private IP while still referring to certificates stored on that machine.
## Certificate generation
Node certificate SAN construction must:
- include every non-empty private node IP;
- include actual public node IPs when present;
- include the selected node address when it is distinct;
- include stable node hostnames;
- include load-balancer addresses returned by the provider;
- omit empty strings;
- deduplicate repeated IPs and names.
Private IP literals used by node URLs must be covered by the generated certificates.
## Public DNS lifecycle
Public DNS records must remain literal public endpoints.
- Create public A records only for VMs with actual public IPs.
- Never publish RFC1918/private addresses into the public roachprod zone.
- A private cluster with no public addresses should produce no public A records.
- `sync` must not recreate public records for private clusters.
- Destroy and DNS cleanup must still remove stale records left by an earlier public incarnation or partial migration.
- Explicit public mode must preserve current public DNS behavior.
## Service discovery
Service-discovery SRV identity must not depend on a public A record existing.
Decouple the SRV target identity from `VM.PublicDNS`, or introduce a node/service identity that is valid for private clusters.
The selected representation must:
- remain stable across `sync`;
- work when no public A record exists;
- preserve current public-cluster discovery;
- avoid accidentally making an internal address publicly resolvable.
Introducing a private DNS zone is not required if service discovery can safely use stable identities or IP-based node resolution.
## Validation and errors
Return clear errors when:
- a requested public URL has no public endpoint;
- a requested private or node URL has no usable address;
- a private multi-network cluster lacks a routed-network declaration;
- certificate generation would otherwise add an empty SAN;
- an SRV target cannot be represented without relying on missing public DNS.
Errors should identify the affected cluster and node.
## Explicitly out of scope
- Migrating roachtest call sites to node endpoints.
- Prometheus, Grafana, Jaeger, and failure-injection address cleanup.
- Creating or configuring private DNS infrastructure.
- Managed-instance-group and load-balancer implementation.
- Roachmgr/Fleetmanager support.
- Changing the default address mode to `auto`.
## Acceptance criteria
- Same-VPC clusters advertise and join through private addresses.
- Public cross-VPC clusters preserve existing public advertise/join behavior.
- Private cross-network clusters fail unless private routing is explicitly declared.
- Routed private cross-network clusters advertise and join through private addresses.
- SQL and Admin UI node URLs work without public IPs or public DNS.
- Explicit public URL requests fail clearly when no public endpoint exists.
- Secure private clusters generate certificates covering all selected private/node addresses.
- Certificate SANs contain no empty or duplicate entries.
- Private clusters create no public A records.
- DNS cleanup removes stale public records even when current VMs have no public IPs.
- SRV discovery works without a public A record.
- Unit tests cover:
- public and private modes;
- same-VPC and multi-VPC topology;
- routed and unrouted private networks;
- secure URL generation;
- certificate SAN construction;
- public DNS create/sync/delete behavior;
- private service discovery.
Jira issue: CRDB-66427
Contributor guide
Research direction
Trace roachprod's address selection, CockroachDB startup and join handling, node URL generation, certificate SAN construction, public DNS lifecycle, and service-discovery paths. Use the acceptance criteria as the test plan: cover public and private modes, network topology, routed networks, secure URLs, certificates, DNS cleanup, and private service discovery, with clear errors for unsupported endpoints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, infrastructure, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100