cockroachdb / cockroachdb/cockroach

roachprod: support multiple separate-process SQL pods per VM on AWS (non-GCE) clusters

Open
#173,265 2 comments 0 reactions 0 assignees View on GitHub
A-roachprod C-enhancement O-agent T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

When benchmarking or testing separate-process (external / `ServiceModeExternal`) SQL tenants, a common need is to run many tenants on a single storage cluster — e.g. multi-tenant noisy-neighbor / isolation experiments with dozens or hundreds of tenants. Each separate-process tenant needs at least one SQL pod (a `cockroach mt start-sql` process), so on a modest cluster you must pack multiple SQL pods onto each VM.

This works on GCE (default project) but **fails on AWS** (and any non-GCE / non-default-GCE-project cluster). Starting a second external SQL instance on a VM fails because roachprod gives it the **same fixed ports** as the first (`--sql-addr=:26257 --http-addr=:26258`), so it can't bind and the process exits (the `cockroach-_.service` systemd unit fails with exit code 1).

Root cause: `SyncedCluster.allowServiceRegistration()` returns `true` only when every VM is GCE **and** in the default GCE project. Per-instance port assignment for separate-process pods happens inside `maybeRegisterServices` → `servicesWithOpenPortSelection` (which finds open ports via `portFunc` starting at `DefaultOpenPortStart`), and that entire path is skipped when registration isn't allowed. With it skipped, external instances fall back to the default 26257/26258 for every instance → port collision. `StartOpts.validate` additionally rejects manually-specified custom ports on such clusters.

**Describe the solution you'd like**

Support multiple separate-process SQL pods per VM on AWS (and other non-GCE clusters). Notably the open-port-finding machinery (`servicesWithOpenPortSelection` / `portFunc`) is already provider-agnostic — only the DNS-SRV-based service *discovery* (`RegisterServices`) is GCE-specific. Decoupling port *assignment* from DNS *registration* — e.g. persist the service→port map in roachprod's local cluster cache and have `pgurl`/service discovery read it on non-GCE clusters — would enable per-instance ports on AWS without Cloud DNS.

**Describe alternatives you've considered**

- Run such experiments on GCE (works today) — not always desirable; AWS parity matters.
- Manually launch `cockroach mt start-sql` with explicit `--sql-addr/--http-addr` ports, bypassing `roachprod start-sql` — works today but loses roachprod's start/stop/monitor/pgurl integration and requires managing certs and systemd units by hand; impractical at the scale of dozens/hundreds of tenants.

**Additional context**

Repro on an AWS roachprod cluster:
```
roachprod start-sql t1 --storage-cluster :1-3 --external-nodes :4
roachprod start-sql t2 --storage-cluster :1-3 --external-nodes :4 --sql-instance 2
```
The second invocation fails; the generated `cockroach-cockroach-t2_2.sh` contains `--sql-addr=:26257 --http-addr=:26258` (identical to t1). roachprod also warns: `WARNING: Service registration and custom ports are not supported for this cluster. ... Attempting to start any additional external SQL processes will fail.`

Key code: [`pkg/roachprod/install/cockroach.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/roachprod/install/cockroach.go) — `allowServiceRegistration`, `maybeRegisterServices`, `servicesWithOpenPortSelection`, `StartOpts.validate`.

Jira issue: CRDB-66587

Contributor guide

Open the contributing guide

Research direction

Start in pkg/roachprod/install/cockroach.go and trace allowServiceRegistration, maybeRegisterServices, servicesWithOpenPortSelection, and StartOpts.validate. Reproduce the two start-sql commands on an AWS cluster and inspect how port assignment and service discovery are handled. Done means multiple external SQL instances can start on one non-GCE VM without port collisions while retaining roachprod lifecycle and pgurl integration.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cli, cloud, databases, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.