cockroachdb / cockroachdb/cockroach

roachprod: make startup, certificates, and DNS address-mode aware

Open
#173,115 1 comment 0 reactions 0 assignees View on GitHub
A-roachprod A-testeng-infra C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Part of #77644.
Follows #173112, #173113, and #173114.

## Summary

Make CockroachDB startup, secure certificates, URL generation, and DNS lifecycle use the cluster's resolved address mode and explicit public/private/node address selectors.

Private clusters must start and remain operable without public IPs or public DNS records. Public-address behavior must remain available for public clusters and tests that deliberately require it.

Design and inventory: https://github.com/cockroachlabs/cockroach/pull/2930

## Motivation

Several roachprod paths still assume that every VM has a public address:

- multi-VPC clusters advertise `Host()`, historically the public IP;
- certificate generation includes `Host()` and load-balancer addresses without consistently checking whether they are present;
- SQL and Admin UI URL generation prefers public DNS and falls back to `PublicIP`;
- service-discovery records use `PublicDNS` as node identity;
- DNS cleanup assumes the current cluster still has public endpoints.

These assumptions can produce empty URLs or certificate SANs, advertise unreachable private addresses across unrouted networks, or publish private addresses into public DNS.

## CockroachDB advertise and join addresses

Node-to-node address selection must depend on network topology and the resolved cluster mode.

### Same routable network

When nodes share a canonical VPC/network identity:

- use private addresses for `--advertise-addr`;
- use private addresses for join targets;
- continue to do so for both public and private allocation modes.

Ordinary multi-region GCE clusters in one VPC fall into this category.

### Different networks

For public-mode clusters spanning different VPCs or providers:

- preserve the existing public advertise/join behavior.

For private-mode clusters spanning different canonical networks:

- require an explicit declaration that the private networks are routed;
- use private addresses when that declaration is present;
- otherwise fail before starting CockroachDB with a clear topology error.

The exact representation of routed-network reachability may be provider metadata or an explicit cluster/start option, but it must not be inferred merely from the presence of private IPs.

## Node-facing URLs

Add node-address support to roachprod URL generation:

- SQL URLs;
- Admin UI URLs;
- other generic node service URLs.

Requirements:

- node URLs use the node selector from #173113;
- explicit public URL requests retain literal public semantics;
- explicit private URL requests retain literal private semantics;
- private clusters do not fall back to an empty public DNS name or public IP;
- certificate-directory selection remains independent of address selection.

For example, a secure node URL used by a process on the invoking machine may use a private IP while still referring to certificates stored on that machine.

## Certificate generation

Node certificate SAN construction must:

- include every non-empty private node IP;
- include actual public node IPs when present;
- include the selected node address when it is distinct;
- include stable node hostnames;
- include load-balancer addresses returned by the provider;
- omit empty strings;
- deduplicate repeated IPs and names.

Private IP literals used by node URLs must be covered by the generated certificates.

## Public DNS lifecycle

Public DNS records must remain literal public endpoints.

- Create public A records only for VMs with actual public IPs.
- Never publish RFC1918/private addresses into the public roachprod zone.
- A private cluster with no public addresses should produce no public A records.
- `sync` must not recreate public records for private clusters.
- Destroy and DNS cleanup must still remove stale records left by an earlier public incarnation or partial migration.
- Explicit public mode must preserve current public DNS behavior.

## Service discovery

Service-discovery SRV identity must not depend on a public A record existing.

Decouple the SRV target identity from `VM.PublicDNS`, or introduce a node/service identity that is valid for private clusters.

The selected representation must:

- remain stable across `sync`;
- work when no public A record exists;
- preserve current public-cluster discovery;
- avoid accidentally making an internal address publicly resolvable.

Introducing a private DNS zone is not required if service discovery can safely use stable identities or IP-based node resolution.

## Validation and errors

Return clear errors when:

- a requested public URL has no public endpoint;
- a requested private or node URL has no usable address;
- a private multi-network cluster lacks a routed-network declaration;
- certificate generation would otherwise add an empty SAN;
- an SRV target cannot be represented without relying on missing public DNS.

Errors should identify the affected cluster and node.

## Explicitly out of scope

- Migrating roachtest call sites to node endpoints.
- Prometheus, Grafana, Jaeger, and failure-injection address cleanup.
- Creating or configuring private DNS infrastructure.
- Managed-instance-group and load-balancer implementation.
- Roachmgr/Fleetmanager support.
- Changing the default address mode to `auto`.

## Acceptance criteria

- Same-VPC clusters advertise and join through private addresses.
- Public cross-VPC clusters preserve existing public advertise/join behavior.
- Private cross-network clusters fail unless private routing is explicitly declared.
- Routed private cross-network clusters advertise and join through private addresses.
- SQL and Admin UI node URLs work without public IPs or public DNS.
- Explicit public URL requests fail clearly when no public endpoint exists.
- Secure private clusters generate certificates covering all selected private/node addresses.
- Certificate SANs contain no empty or duplicate entries.
- Private clusters create no public A records.
- DNS cleanup removes stale public records even when current VMs have no public IPs.
- SRV discovery works without a public A record.
- Unit tests cover:
- public and private modes;
- same-VPC and multi-VPC topology;
- routed and unrouted private networks;
- secure URL generation;
- certificate SAN construction;
- public DNS create/sync/delete behavior;
- private service discovery.

Jira issue: CRDB-66427

Contributor guide

Open the contributing guide

Research direction

Trace roachprod's address selection, CockroachDB startup and join handling, node URL generation, certificate SAN construction, public DNS lifecycle, and service-discovery paths. Use the acceptance criteria as the test plan: cover public and private modes, network topology, routed networks, secure URLs, certificates, DNS cleanup, and private service discovery, with clear errors for unsupported endpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, infrastructure, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.