cockroachdb / cockroachdb/cockroach
roachprod: managed private GCE VMs lose IAP transport metadata during create and grow
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Describe the problem
Creating a managed GCE cluster with private addresses and IAP provisions the VM correctly with the `iap-ssh` network tag, but roachprod drops that tag when converting managed-instance-group data into `vm.VM`.
Because `gce.UsesIAP` relies exclusively on `vm.VM.NetworkTags`, initial SSH attempts connect directly to the private IP instead of using `gcloud compute start-iap-tunnel`.
Affected configuration:
```bash
roachprod create "$USER-iap-mig" \
--nodes=1 \
--clouds=gce \
--gce-managed \
--address-mode=private \
--gce-use-iap \
--keep-on-failure
```
This was found by static analysis of #3302 and has not been reproduced against a live GCE project.
## Root cause
`computeAddressArgs` correctly creates managed instance templates with:
```text
--no-address --tags iap-ssh
```
However, `jsonInstanceTemplate.Properties` in `pkg/roachprod/vm/gce/gcloud.go` does not model `properties.tags`. Consequently, `managedInstanceGroupInstance.toVM` does not populate `vm.VM.NetworkTags`.
The resulting cached VM resembles:
```text
PublicIP: ""
PrivateIP: "10.x.x.x"
AddressMode: private
NetworkTags: []
```
`gce.UsesIAP` therefore returns false. `sshNetworkArgs` omits the IAP `ProxyCommand`, and `Host` selects the private IP.
The create path saves this lossy VM representation and immediately calls:
```go
SetupSSH(ctx, l, clusterName, false /* sync */)
```
Because no cloud sync occurs first, the actual instance tags are not recovered before SSH.
Standalone CLI and SDK conversion paths copy network tags correctly; the problem is specific to the immediate managed-instance-group create/grow path.
## Expected behavior
Managed private VMs created with `--gce-use-iap` retain `iap-ssh` in `vm.VM.NetworkTags`, and initial SSH/SCP uses:
```text
gcloud compute start-iap-tunnel ...
```
## Actual behavior
Roachprod attempts a direct SSH connection to the VM's private address. For callers outside the VPC, this normally times out.
During create, that error can cause roachprod to report failure and delete the successfully provisioned cluster unless `--keep-on-failure` is used. Grow can similarly report failure after adding the new instances.
A later `roachprod sync` can repair the cached representation because ordinary instance-list conversion preserves tags, but initial setup has already failed.
## Suggested fix
Add template tags to `jsonInstanceTemplate.Properties` and copy them into the constructed VM:
```go
NetworkTags: instanceTemplate.Properties.Tags.Items,
```
## Regression test
Extend `TestManagedInstanceToVMPrivate` to:
1. Unmarshal an instance template containing `properties.tags.items=["iap-ssh"]`.
2. Convert it with `managedInstanceGroupInstance.toVM`.
3. Assert that `v.NetworkTags` contains `iap-ssh`.
4. Assert `gce.UsesIAP(*v)` is true.
5. Verify the corresponding SSH network arguments contain `gcloud compute start-iap-tunnel`.
Jira issue: CRDB-66634
Contributor guide
Research direction
Start in pkg/roachprod/vm/gce/gcloud.go, reading jsonInstanceTemplate.Properties and managedInstanceGroupInstance.toVM, then run the existing TestManagedInstanceToVMPrivate. Verify the managed template's iap-ssh tag reaches vm.VM.NetworkTags, gce.UsesIAP returns true, and the SSH network arguments include gcloud compute start-iap-tunnel.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, go
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100