cockroachdb / cockroachdb/cockroach

roachprod: managed private GCE VMs lose IAP transport metadata during create and grow

Open
#173,316 2 comments 0 reactions 0 assignees View on GitHub
A-testeng-infra branch-master C-bug O-agent T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Describe the problem

Creating a managed GCE cluster with private addresses and IAP provisions the VM correctly with the `iap-ssh` network tag, but roachprod drops that tag when converting managed-instance-group data into `vm.VM`.

Because `gce.UsesIAP` relies exclusively on `vm.VM.NetworkTags`, initial SSH attempts connect directly to the private IP instead of using `gcloud compute start-iap-tunnel`.

Affected configuration:

```bash
roachprod create "$USER-iap-mig" \
--nodes=1 \
--clouds=gce \
--gce-managed \
--address-mode=private \
--gce-use-iap \
--keep-on-failure
```

This was found by static analysis of #3302 and has not been reproduced against a live GCE project.

## Root cause

`computeAddressArgs` correctly creates managed instance templates with:

```text
--no-address --tags iap-ssh
```

However, `jsonInstanceTemplate.Properties` in `pkg/roachprod/vm/gce/gcloud.go` does not model `properties.tags`. Consequently, `managedInstanceGroupInstance.toVM` does not populate `vm.VM.NetworkTags`.

The resulting cached VM resembles:

```text
PublicIP: ""
PrivateIP: "10.x.x.x"
AddressMode: private
NetworkTags: []
```

`gce.UsesIAP` therefore returns false. `sshNetworkArgs` omits the IAP `ProxyCommand`, and `Host` selects the private IP.

The create path saves this lossy VM representation and immediately calls:

```go
SetupSSH(ctx, l, clusterName, false /* sync */)
```

Because no cloud sync occurs first, the actual instance tags are not recovered before SSH.

Standalone CLI and SDK conversion paths copy network tags correctly; the problem is specific to the immediate managed-instance-group create/grow path.

## Expected behavior

Managed private VMs created with `--gce-use-iap` retain `iap-ssh` in `vm.VM.NetworkTags`, and initial SSH/SCP uses:

```text
gcloud compute start-iap-tunnel ...
```

## Actual behavior

Roachprod attempts a direct SSH connection to the VM's private address. For callers outside the VPC, this normally times out.

During create, that error can cause roachprod to report failure and delete the successfully provisioned cluster unless `--keep-on-failure` is used. Grow can similarly report failure after adding the new instances.

A later `roachprod sync` can repair the cached representation because ordinary instance-list conversion preserves tags, but initial setup has already failed.

## Suggested fix

Add template tags to `jsonInstanceTemplate.Properties` and copy them into the constructed VM:

```go
NetworkTags: instanceTemplate.Properties.Tags.Items,
```

## Regression test

Extend `TestManagedInstanceToVMPrivate` to:

1. Unmarshal an instance template containing `properties.tags.items=["iap-ssh"]`.
2. Convert it with `managedInstanceGroupInstance.toVM`.
3. Assert that `v.NetworkTags` contains `iap-ssh`.
4. Assert `gce.UsesIAP(*v)` is true.
5. Verify the corresponding SSH network arguments contain `gcloud compute start-iap-tunnel`.

Jira issue: CRDB-66634

Contributor guide

Open the contributing guide

Research direction

Start in pkg/roachprod/vm/gce/gcloud.go, reading jsonInstanceTemplate.Properties and managedInstanceGroupInstance.toVM, then run the existing TestManagedInstanceToVMPrivate. Verify the managed template's iap-ssh tag reaches vm.VM.NetworkTags, gce.UsesIAP returns true, and the SSH network arguments include gcloud compute start-iap-tunnel.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.