cockroachdb / cockroachdb/cockroach
roachprod: cannot pack multiple different separate-process tenants on one VM (port assignment is blind across tenants)
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Benchmarking many separate-process (external / `ServiceModeExternal`) tenants — e.g. a multi-tenant isolation experiment with one large tenant plus 100 small tenants — requires packing multiple **different** tenants' SQL pods onto each VM. `roachprod start-sql` cannot do this:
- Multiple **instances of the same tenant** on a node work fine — they get distinct offset ports (e.g. `29000/29001`, `29002/29003`).
- Multiple **different tenants** on the same node all get assigned the **same** ports (`--sql-addr=:29000 --http-addr=:29001`), so the second and later tenants fail to bind:
`server startup failed: ... listen tcp :29001: bind: address already in use`.
This reproduces on **GCE (default project, service registration enabled)**, so it is distinct from #173265 (which is about AWS / non-GCE clusters lacking service registration entirely). Here registration *is* on and per-instance port selection runs — it just doesn't account for other tenants.
**Repro (GCE cluster):**
```
roachprod start-sql t1 --storage-cluster :1-3 --external-nodes :5
roachprod start-sql t2 --storage-cluster :1-3 --external-nodes :5 --sql-instance 2
```
The second pod fails; its generated command shows `--sql-addr=:29000 --http-addr=:29001` (identical to t1). Running them **serially** (waiting for t1 to come up first) collides the same way. Passing `--sql-port` only sets the SQL port; the HTTP port is still auto-scanned and collides.
**Root-cause hypothesis:** port selection in `maybeRegisterServices` → `servicesWithOpenPortSelection` (`pkg/roachprod/install/cockroach.go`) derives ports from `MapServices` scoped to the *given* tenant, so a new tenant restarts the search at `DefaultOpenPortStart` (29000) and does not consider ports already bound by *other* tenants' services on the same node.
**Describe the solution you'd like**
Assign distinct ports across **all** services on a node regardless of tenant, so many different separate-process tenants can be packed per VM. Alternatively (or additionally), expose an `--http-port` flag alongside `--sql-port` so callers can pre-assign non-colliding ports explicitly.
**Describe alternatives you've considered**
Bypass `roachprod start-sql` and launch `cockroach mt start-sql` directly with explicit `--sql-addr` **and** `--http-addr` per tenant. This works and packs cleanly, but loses roachprod's start/stop/monitor/`pgurl` integration and requires manually replicating tenant creation (`CREATE TENANT` / `ALTER TENANT … START SERVICE EXTERNAL`) and cert minting (`cert create-tenant-client`, `mt cert create-tenant-signing`).
**Additional context**
Related: #173265 (multi-pod-per-VM on AWS / non-GCE clusters — service registration gated to the default GCE project). That issue is about registration being disabled entirely off-GCE; this one is about the on-GCE port assignment not being tenant-aware. A fix that makes port assignment consider all node-local services would help both.
Key code: [`pkg/roachprod/install/cockroach.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/roachprod/install/cockroach.go) — `maybeRegisterServices`, `servicesWithOpenPortSelection`, `MapServices`.
Jira issue: CRDB-66649
Contributor guide
Research direction
Start in pkg/roachprod/install/cockroach.go, reading maybeRegisterServices, servicesWithOpenPortSelection, and MapServices. Reproduce the GCE scenario with the two roachprod start-sql commands and inspect the generated addresses. Done means different separate-process tenants receive non-colliding SQL and HTTP ports on the same node, or explicit HTTP port assignment is also provided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100