cockroachdb / cockroachdb/cockroach

roachtest: make the classic roachprod backend private-address compatible

Open
#173,116 1 comment 0 reactions 0 assignees View on GitHub
A-roachprod A-testeng-infra A-testing C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Part of #77644.
Follows #173112, #173113, #173114, and #173115.

## Summary

Make roachtest clusters using the classic roachprod backend operate without public VM addresses.

This includes:

- propagating `--address-mode=auto|public|private`;
- using node endpoints for runner-to-node connections;
- preserving literal public and private endpoint semantics;
- removing public-IP assumptions from monitoring;
- making network failure injection work when public addresses are absent.

Roachmgr/Fleetmanager compatibility is tracked separately.

Design and inventory: https://github.com/cockroachlabs/cockroach/pull/2930

## Motivation

Roachprod can provision private GCE VMs, but roachtest still assumes public endpoints in several independent paths:

- roachtest cluster creation does not propagate address mode;
- `ConnE` always builds an external/public PG URL;
- runner-side Admin UI and debug clients request external endpoints;
- runner-side test services use `ExternalIP`;
- local and centralized Prometheus may select public node addresses;
- Grafana, Jaeger, and clusterstats use external endpoints;
- network partition and latency helpers unconditionally expand public addresses.

Consequently, private VMs may be reachable from the runner while ordinary roachtests still fail because `PublicIP` is empty.

## Address-mode propagation

Add a roachtest run flag:

--address-mode=auto|public|private

Initial behavior:

- the default remains `public`;
- explicit values are passed through `vm.CreateOpts`;
- CRDB and workload node groups receive the same requested mode;
- create retries preserve the requested mode;
- cluster-spec resolution preserves the requested mode;
- provider resolution and unsupported-provider validation remain owned by roachprod.

For reused or explicitly supplied clusters:

- use the persisted resolved mode from #173112;
- do not re-evaluate `auto`;
- reject reuse when an explicitly requested mode is incompatible with the existing cluster.

## Endpoint semantics

Expose explicit internal, external, and node endpoint APIs through the roachtest cluster interface.

### Internal endpoints

Internal endpoints retain private, node-to-node semantics.

URLs intended for commands executed on cluster nodes use node-side certificate paths.

### External endpoints

External endpoints retain literal public semantics.

They return a clear error when the node has no public endpoint.

### Node endpoints

Node endpoints use the node-address selector from #173113 and are intended for connections from the roachtest runner.

For secure clusters, runner-side node URLs use runner-local certificate paths.

Address selection and certificate location must remain independent.

Possible interface methods include:

NodeIP(...)
NodeAddr(...)
NodePGUrl(...)
NodeAdminUIAddr(...)

The exact names are an implementation detail.

## Runner-side migration

Migrate runner-side consumers from external endpoints to node endpoints, including:

- `roachprodCluster.ConnE`;
- direct SQL and PGX clients;
- Admin UI and debug-zip collection;
- profile and diagnostics helpers;
- runner access to test-local HTTP, SQL, proxy, webhook, and metrics services;
- operator-facing node URLs emitted by roachtest.

Representative affected tests and helpers include:

- Rust SQLx;
- backup and restore;
- cluster-to-cluster;
- drain and gossip;
- status server;
- Kerberos and LDAP;
- CDC;
- connection latency;
- vectorized SQL tests;
- clusterstats and profile helpers.

## Call-site classification

Audit existing `ExternalPGUrl`, `ExternalAdminUIAddr`, `ExternalAddr`, and `ExternalIP` uses.

Classify each use as:

- runner-to-node: migrate to a node endpoint;
- node-to-node: migrate to an internal/private endpoint;
- genuinely public: retain an explicit external/public endpoint.

Direct `ExternalIP` uses in CDC, backup, connection-latency, export, gossip, LDAP, network, and TPCC tests require individual classification.

Tests involving genuinely external systems must remain explicitly public unless a private interconnect is part of the test environment.

Cloud/project checks must not be duplicated at call sites.

## Prometheus and exporter access

Update local Prometheus behavior:

- scrape cluster nodes through private addresses;
- use the Prometheus node's private source address in exporter firewall rules;
- do not expand `{ip:...:public}` for private-only nodes;
- preserve public-mode compatibility.

Update centralized Prometheus behavior:

- select private targets only when the monitoring service has routed access;
- make routed monitoring reachability explicit rather than inferring it from a project name where possible;
- never silently fall back to an empty public address;
- emit a clear warning or disable registration when no reachable target exists.

Parameterize exporter firewall source CIDRs rather than relying only on the hard-coded `10.0.0.0/8` range and public DNS results.

## Grafana, Jaeger, and metrics helpers

Use node endpoints for runner-side access to:

- Grafana;
- Jaeger;
- clusterstats;
- TPCC chaos metrics;
- test-local Prometheus;
- other node-hosted diagnostics services.

Node-to-node monitoring traffic must use private addresses.

## Failure injection

Make network failure helpers work when public addresses are absent.

For network partition and latency injection:

- private-address rules remain required;
- public-address rules are installed only when actual public addresses exist;
- helpers must not weaken the literal behavior of `{ip:...:public}`;
- cleanup must mirror the rules that were installed;
- public-mode behavior remains unchanged.

Address-semantics tests, including gossip locality-address checks, must derive their expected behavior from the resolved mode or provide explicit public/private variants.

## Interface and test infrastructure

Update:

- the roachtest cluster interface;
- classic roachprod-backed implementation;
- generated mocks;
- fake/test cluster implementations;
- API comments documenting endpoint and certificate semantics.

The roachmgr implementation may return an explicit unsupported error until its separate private-address issue is completed.

## Error behavior

Return clear errors when:

- an explicit public endpoint is requested for a private-only node;
- no node address is available;
- a secure runner-side URL lacks usable runner-local certificates;
- monitoring has neither private reachability nor a public endpoint;
- cluster reuse conflicts with an explicitly requested address mode.

Errors should identify the cluster, node, and requested endpoint kind.

## Explicitly out of scope

- Roachmgr/Fleetmanager private-address support.
- Managed instance groups and internal load balancers.
- Creating VPCs, routes, NAT, or firewall infrastructure.
- Migrating GCP buckets, identities, Pub/Sub, KMS, or snapshots.
- Changing the default address mode from `public` to `auto`.

## Acceptance criteria

- Roachtest accepts and propagates `--address-mode=auto|public|private`.
- Split CRDB/workload clusters and retries preserve the requested mode.
- Cluster reuse respects the persisted resolved mode.
- The cluster interface exposes explicit internal, external, and node endpoint semantics.
- `ConnE` works from the runner against private-only secure and insecure clusters.
- Runner-side SQL, Admin UI, debug, profile, and test-service clients use node endpoints.
- Node-to-node services use internal/private endpoints.
- Explicit external APIs fail clearly when no public endpoint exists.
- Local Prometheus scrapes private node addresses.
- Exporter firewall rules work without public IPs and with configured private CIDRs.
- Centralized monitoring never registers an empty or unreachable public target.
- Grafana, Jaeger, clusterstats, and metrics helpers work through node endpoints.
- Network partition and latency injection work on private-only clusters.
- Genuine public-connectivity tests retain explicit public endpoints.
- Unit tests cover flag propagation, endpoint selection, certificate paths, monitoring, failure injection, and cluster reuse.
- Private-address smoke coverage includes:
- a runner SQL test such as `rust/sqlx`;
- an Admin UI test such as `status-server`;
- a monitoring-enabled test;
- a network failure-injection test.

Jira issue: CRDB-66428

Contributor guide

Open the contributing guide

Research direction

Start with the design and inventory in PR 2930, then trace the roachtest cluster interface and roachprodCluster.ConnE before auditing ExternalPGUrl, ExternalAdminUIAddr, ExternalAddr, and ExternalIP call sites. Validate the acceptance criteria with unit coverage and the named rust/sqlx, status-server, monitoring, and network failure-injection smoke scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go, prometheus
Domain
infrastructure, networking, observability-sre, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.