cockroachdb / cockroachdb/cockroach
roachprod: support multiple separate-process SQL pods per VM on AWS (non-GCE) clusters
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
When benchmarking or testing separate-process (external / `ServiceModeExternal`) SQL tenants, a common need is to run many tenants on a single storage cluster — e.g. multi-tenant noisy-neighbor / isolation experiments with dozens or hundreds of tenants. Each separate-process tenant needs at least one SQL pod (a `cockroach mt start-sql` process), so on a modest cluster you must pack multiple SQL pods onto each VM.
This works on GCE (default project) but **fails on AWS** (and any non-GCE / non-default-GCE-project cluster). Starting a second external SQL instance on a VM fails because roachprod gives it the **same fixed ports** as the first (`--sql-addr=:26257 --http-addr=:26258`), so it can't bind and the process exits (the `cockroach-_.service` systemd unit fails with exit code 1).
Root cause: `SyncedCluster.allowServiceRegistration()` returns `true` only when every VM is GCE **and** in the default GCE project. Per-instance port assignment for separate-process pods happens inside `maybeRegisterServices` → `servicesWithOpenPortSelection` (which finds open ports via `portFunc` starting at `DefaultOpenPortStart`), and that entire path is skipped when registration isn't allowed. With it skipped, external instances fall back to the default 26257/26258 for every instance → port collision. `StartOpts.validate` additionally rejects manually-specified custom ports on such clusters.
**Describe the solution you'd like**
Support multiple separate-process SQL pods per VM on AWS (and other non-GCE clusters). Notably the open-port-finding machinery (`servicesWithOpenPortSelection` / `portFunc`) is already provider-agnostic — only the DNS-SRV-based service *discovery* (`RegisterServices`) is GCE-specific. Decoupling port *assignment* from DNS *registration* — e.g. persist the service→port map in roachprod's local cluster cache and have `pgurl`/service discovery read it on non-GCE clusters — would enable per-instance ports on AWS without Cloud DNS.
**Describe alternatives you've considered**
- Run such experiments on GCE (works today) — not always desirable; AWS parity matters.
- Manually launch `cockroach mt start-sql` with explicit `--sql-addr/--http-addr` ports, bypassing `roachprod start-sql` — works today but loses roachprod's start/stop/monitor/pgurl integration and requires managing certs and systemd units by hand; impractical at the scale of dozens/hundreds of tenants.
**Additional context**
Repro on an AWS roachprod cluster:
```
roachprod start-sql t1 --storage-cluster :1-3 --external-nodes :4
roachprod start-sql t2 --storage-cluster :1-3 --external-nodes :4 --sql-instance 2
```
The second invocation fails; the generated `cockroach-cockroach-t2_2.sh` contains `--sql-addr=:26257 --http-addr=:26258` (identical to t1). roachprod also warns: `WARNING: Service registration and custom ports are not supported for this cluster. ... Attempting to start any additional external SQL processes will fail.`
Key code: [`pkg/roachprod/install/cockroach.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/roachprod/install/cockroach.go) — `allowServiceRegistration`, `maybeRegisterServices`, `servicesWithOpenPortSelection`, `StartOpts.validate`.
Jira issue: CRDB-66587
Contributor guide
Research direction
Start in pkg/roachprod/install/cockroach.go and trace allowServiceRegistration, maybeRegisterServices, servicesWithOpenPortSelection, and StartOpts.validate. Reproduce the two start-sql commands on an AWS cluster and inspect how port assignment and service discovery are handled. Done means multiple external SQL instances can start on one non-GCE VM without port collisions while retaining roachprod lifecycle and pgurl integration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cli, cloud, databases, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100