cockroachdb / cockroachdb/cockroach

roachtest roachmgr backend: support private node endpoints

Open
#173,119 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-infra A-testing C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Part of #77644.
Follows #173112, #173113, and #173116.
Blocked on https://github.com/cockroachlabs/roachmgr/issues/1091.

## Summary

Make the roachtest roachmgr backend create and operate clusters without public machine addresses.

Consume the upstream roachmgr SDK address mode, propagate the shared roachtest setting, and implement node endpoint selection using realized private machine state.

Design and inventory:
https://github.com/cockroachlabs/cockroach/pull/2930

## Motivation

The roachmgr adapter already records both private and public machine addresses, but its runner-facing paths require public state:

- external SQL URLs use `PublicSQLAddr`;
- external addresses require `Machine.PublicAddr` and `PublicPorts`;
- `ConnE` builds a public SQL URL;
- Admin UI helpers require public addresses;
- start validates that every CRDB node has a usable public SQL address;
- centralized monitoring may select public targets.

A private roachmgr cluster can therefore be provisioned successfully but still fail before an ordinary roachtest begins.

## SDK integration

Upgrade to the roachmgr SDK version that supports resolved public/private address mode.

Propagate the shared roachtest setting:

--address-mode=auto|public|private

Behavior:

- resolve `auto` using the same provider/project policy as the classic backend;
- render the resolved `public` or `private` value into roachmgr infra;
- apply the same mode to CRDB and workload machine groups;
- preserve the mode across retries and state reloads;
- retain `public` as the rollout default.

Do not infer or override the mode from `network-access=direct|proxy`.

## Node endpoint implementation

Implement the node endpoint APIs introduced by #173116 for `roachmgrCluster`.

### Private mode

- `NodeIP` uses `Machine.PrivateIP`;
- node SQL and Admin UI addresses use private service addresses and original service ports;
- `PublicPorts` are not required;
- runner-side secure URLs use runner-local certificates.

### Public mode

- node endpoints preserve the existing public address and port-mapping behavior;
- explicit external APIs continue to return literal public endpoints.

### Explicit endpoint semantics

- internal endpoints remain private node-to-node endpoints;
- external endpoints remain literal public endpoints;
- node endpoints follow the resolved mode.

Missing requested addresses must return clear errors.

## SQL connectivity

Replace `validateRoachmgrPublicSQLConnectivity` with validation of the selected node SQL address.

Update:

- `ConnE`;
- `ExternalPGUrl` and the new node PG URL API;
- SQL address helpers;
- realized CockroachDB state handling.

A private node SQL address must use the runner-local certificate directory when consumed by the roachtest runner.

## Admin UI and runner services

Implement private node addressing for:

- Admin UI access;
- debug and profile collection;
- runner-side test services;
- operator-facing node URLs.

Do not require a public port mapping for services reached directly through private routing.

## Command execution

Roachmgr command execution may continue through its existing infra-exec mechanism.

Address mode must not change:

- artifact upload/download;
- service control;
- diagnostics collection;
- failure injection through infra-exec.

State and artifact snapshots must tolerate empty public address fields.

## Monitoring

Update roachmgr centralized-Prometheus target selection:

- use private targets when monitoring has routed reachability;
- retain public targets for environments without private monitoring routes;
- never register an empty public address;
- keep address visibility separate from direct/proxy command transport.

## Validation

Return clear errors when:

- private mode is requested with an SDK/provider that does not support it;
- realized private state lacks a private address;
- the runner has no declared private route for direct node endpoints;
- an explicit public endpoint is requested from a private-only machine;
- CRDB and workload groups resolve to incompatible modes.

## Explicitly out of scope

- Implementing address visibility inside Fleetmanager providers.
- Classic roachprod backend compatibility.
- Provisioning private VPC infrastructure.
- Managed-instance-group load balancers.
- Changing the default from `public` to `auto`.

## Acceptance criteria

- Cockroach consumes the released address-mode-capable roachmgr SDK.
- The shared roachtest address-mode flag is rendered into roachmgr infra.
- CRDB and workload groups receive the same resolved mode.
- Private roachmgr machines have no public IPs or public port mappings.
- Start no longer requires a public SQL address.
- `ConnE` works against a private secure cluster using runner-local certificates.
- Node SQL and Admin UI endpoints use private addresses and service ports.
- Explicit external APIs fail clearly when public state is absent.
- Command execution, artifacts, diagnostics, and service control continue working.
- Centralized monitoring uses a reachable private target or disables registration clearly.
- Unit tests cover mode propagation, state reload, private/public endpoint selection, certificate paths, and missing public state.
- A private-address roachmgr smoke test creates, starts, connects to, collects artifacts from, and destroys a cluster.

Jira issue: CRDB-66431

Contributor guide

Open the contributing guide

Research direction

Start with the roachmgrCluster node endpoint APIs introduced by #173116, then trace address-mode propagation into ConnE, ExternalPGUrl, validateRoachmgrPublicSQLConnectivity, and centralized monitoring. Use the acceptance criteria to define done: tests cover mode propagation, state reload, endpoint selection, certificate paths, and missing public state, while a private-address smoke test covers cluster lifecycle and artifacts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems, networking, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
22/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.