cockroachdb / cockroachdb/cockroach
roachtest roachmgr backend: support private node endpoints
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Part of #77644.
Follows #173112, #173113, and #173116.
Blocked on https://github.com/cockroachlabs/roachmgr/issues/1091.
## Summary
Make the roachtest roachmgr backend create and operate clusters without public machine addresses.
Consume the upstream roachmgr SDK address mode, propagate the shared roachtest setting, and implement node endpoint selection using realized private machine state.
Design and inventory:
https://github.com/cockroachlabs/cockroach/pull/2930
## Motivation
The roachmgr adapter already records both private and public machine addresses, but its runner-facing paths require public state:
- external SQL URLs use `PublicSQLAddr`;
- external addresses require `Machine.PublicAddr` and `PublicPorts`;
- `ConnE` builds a public SQL URL;
- Admin UI helpers require public addresses;
- start validates that every CRDB node has a usable public SQL address;
- centralized monitoring may select public targets.
A private roachmgr cluster can therefore be provisioned successfully but still fail before an ordinary roachtest begins.
## SDK integration
Upgrade to the roachmgr SDK version that supports resolved public/private address mode.
Propagate the shared roachtest setting:
--address-mode=auto|public|private
Behavior:
- resolve `auto` using the same provider/project policy as the classic backend;
- render the resolved `public` or `private` value into roachmgr infra;
- apply the same mode to CRDB and workload machine groups;
- preserve the mode across retries and state reloads;
- retain `public` as the rollout default.
Do not infer or override the mode from `network-access=direct|proxy`.
## Node endpoint implementation
Implement the node endpoint APIs introduced by #173116 for `roachmgrCluster`.
### Private mode
- `NodeIP` uses `Machine.PrivateIP`;
- node SQL and Admin UI addresses use private service addresses and original service ports;
- `PublicPorts` are not required;
- runner-side secure URLs use runner-local certificates.
### Public mode
- node endpoints preserve the existing public address and port-mapping behavior;
- explicit external APIs continue to return literal public endpoints.
### Explicit endpoint semantics
- internal endpoints remain private node-to-node endpoints;
- external endpoints remain literal public endpoints;
- node endpoints follow the resolved mode.
Missing requested addresses must return clear errors.
## SQL connectivity
Replace `validateRoachmgrPublicSQLConnectivity` with validation of the selected node SQL address.
Update:
- `ConnE`;
- `ExternalPGUrl` and the new node PG URL API;
- SQL address helpers;
- realized CockroachDB state handling.
A private node SQL address must use the runner-local certificate directory when consumed by the roachtest runner.
## Admin UI and runner services
Implement private node addressing for:
- Admin UI access;
- debug and profile collection;
- runner-side test services;
- operator-facing node URLs.
Do not require a public port mapping for services reached directly through private routing.
## Command execution
Roachmgr command execution may continue through its existing infra-exec mechanism.
Address mode must not change:
- artifact upload/download;
- service control;
- diagnostics collection;
- failure injection through infra-exec.
State and artifact snapshots must tolerate empty public address fields.
## Monitoring
Update roachmgr centralized-Prometheus target selection:
- use private targets when monitoring has routed reachability;
- retain public targets for environments without private monitoring routes;
- never register an empty public address;
- keep address visibility separate from direct/proxy command transport.
## Validation
Return clear errors when:
- private mode is requested with an SDK/provider that does not support it;
- realized private state lacks a private address;
- the runner has no declared private route for direct node endpoints;
- an explicit public endpoint is requested from a private-only machine;
- CRDB and workload groups resolve to incompatible modes.
## Explicitly out of scope
- Implementing address visibility inside Fleetmanager providers.
- Classic roachprod backend compatibility.
- Provisioning private VPC infrastructure.
- Managed-instance-group load balancers.
- Changing the default from `public` to `auto`.
## Acceptance criteria
- Cockroach consumes the released address-mode-capable roachmgr SDK.
- The shared roachtest address-mode flag is rendered into roachmgr infra.
- CRDB and workload groups receive the same resolved mode.
- Private roachmgr machines have no public IPs or public port mappings.
- Start no longer requires a public SQL address.
- `ConnE` works against a private secure cluster using runner-local certificates.
- Node SQL and Admin UI endpoints use private addresses and service ports.
- Explicit external APIs fail clearly when public state is absent.
- Command execution, artifacts, diagnostics, and service control continue working.
- Centralized monitoring uses a reachable private target or disables registration clearly.
- Unit tests cover mode propagation, state reload, private/public endpoint selection, certificate paths, and missing public state.
- A private-address roachmgr smoke test creates, starts, connects to, collects artifacts from, and destroys a cluster.
Jira issue: CRDB-66431
Contributor guide
Research direction
Start with the roachmgrCluster node endpoint APIs introduced by #173116, then trace address-mode propagation into ConnE, ExternalPGUrl, validateRoachmgrPublicSQLConnectivity, and centralized monitoring. Use the acceptance criteria to define done: tests cover mode propagation, state reload, endpoint selection, certificate paths, and missing public state, while a private-address smoke test covers cluster lifecycle and artifacts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems, networking, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 22/100