cockroachdb / cockroachdb/cockroach

roachprod: cannot pack multiple different separate-process tenants on one VM (port assignment is blind across tenants)

Open
#173,330 1 comment 0 reactions 0 assignees View on GitHub
A-roachprod C-enhancement O-agent T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

Benchmarking many separate-process (external / `ServiceModeExternal`) tenants — e.g. a multi-tenant isolation experiment with one large tenant plus 100 small tenants — requires packing multiple **different** tenants' SQL pods onto each VM. `roachprod start-sql` cannot do this:

- Multiple **instances of the same tenant** on a node work fine — they get distinct offset ports (e.g. `29000/29001`, `29002/29003`).
- Multiple **different tenants** on the same node all get assigned the **same** ports (`--sql-addr=:29000 --http-addr=:29001`), so the second and later tenants fail to bind:
`server startup failed: ... listen tcp :29001: bind: address already in use`.

This reproduces on **GCE (default project, service registration enabled)**, so it is distinct from #173265 (which is about AWS / non-GCE clusters lacking service registration entirely). Here registration *is* on and per-instance port selection runs — it just doesn't account for other tenants.

**Repro (GCE cluster):**
```
roachprod start-sql t1 --storage-cluster :1-3 --external-nodes :5
roachprod start-sql t2 --storage-cluster :1-3 --external-nodes :5 --sql-instance 2
```
The second pod fails; its generated command shows `--sql-addr=:29000 --http-addr=:29001` (identical to t1). Running them **serially** (waiting for t1 to come up first) collides the same way. Passing `--sql-port` only sets the SQL port; the HTTP port is still auto-scanned and collides.

**Root-cause hypothesis:** port selection in `maybeRegisterServices` → `servicesWithOpenPortSelection` (`pkg/roachprod/install/cockroach.go`) derives ports from `MapServices` scoped to the *given* tenant, so a new tenant restarts the search at `DefaultOpenPortStart` (29000) and does not consider ports already bound by *other* tenants' services on the same node.

**Describe the solution you'd like**

Assign distinct ports across **all** services on a node regardless of tenant, so many different separate-process tenants can be packed per VM. Alternatively (or additionally), expose an `--http-port` flag alongside `--sql-port` so callers can pre-assign non-colliding ports explicitly.

**Describe alternatives you've considered**

Bypass `roachprod start-sql` and launch `cockroach mt start-sql` directly with explicit `--sql-addr` **and** `--http-addr` per tenant. This works and packs cleanly, but loses roachprod's start/stop/monitor/`pgurl` integration and requires manually replicating tenant creation (`CREATE TENANT` / `ALTER TENANT … START SERVICE EXTERNAL`) and cert minting (`cert create-tenant-client`, `mt cert create-tenant-signing`).

**Additional context**

Related: #173265 (multi-pod-per-VM on AWS / non-GCE clusters — service registration gated to the default GCE project). That issue is about registration being disabled entirely off-GCE; this one is about the on-GCE port assignment not being tenant-aware. A fix that makes port assignment consider all node-local services would help both.

Key code: [`pkg/roachprod/install/cockroach.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/roachprod/install/cockroach.go) — `maybeRegisterServices`, `servicesWithOpenPortSelection`, `MapServices`.

Jira issue: CRDB-66649

Contributor guide

Open the contributing guide

Research direction

Start in pkg/roachprod/install/cockroach.go, reading maybeRegisterServices, servicesWithOpenPortSelection, and MapServices. Reproduce the GCE scenario with the two roachprod start-sql commands and inspect the generated addresses. Done means different separate-process tenants receive non-colliding SQL and HTTP ports on the same node, or explicit HTTP port assignment is also provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.