hashicorp / hashicorp/packer-plugin-googlecompute
IAP Cloud Build: SSH Permission Denied
- Dominant language
- Go
- Stars
- 33
- Forks
- 72
- Avg merge
- 6h 47m
- Merged PRs (30d)
- 3
Description
#### Overview of the Issue
I am trying to use Google Cloud Build to create Packer images, but with the machines being configured to have no external IP. The IAP options seem perfect for this, but I have yet to get a standard build to finish with those options being set. It always fails when trying to establish the SSH connection.
I have verified that it is not a permissions issue with the firewall rules or the service account IAM permissions as I can impersonate it on the CLI and there is no problem connecting through SSH to the private machines.
#### Reproduction Steps
1. Prepare a Google Cloud Build run, with the relevant steps to execute Packer with this plugin (example below).
2. Supply Packer with the config file described below (I do it through a cloud source repo).
3. Assign the proper permissions to the cloud build service account (I use a custom one). Most notable is `IAP-secured Tunnel User` but others will probably be required.
4. Configure your VPC firewall rules to allow the IAP CIDR (35.235.240.0/20) into all the instances in the VPC.
5. Trigger the build step and see it hang and timeout on this message: "Waiting for SSH to become available...".
### Plugin and Packer version
Plugin: 1.0.13
Packer: Docker image, light-1.8.2
### Simplified Packer Buildfile
Note: I have tried with a dynamic and static IAP localhost port, with no setting making a difference.
```hcl
source "googlecompute" "base-deb11" {
source_image = "a custom Debian 11 image"
enable_secure_boot = true
enable_vtpm = true
enable_integrity_monitoring = true
network = "network"
subnetwork = "subnet"
omit_external_ip = true
use_internal_ip = true
use_iap = true
# iap_localhost_port = 22
service_account_email = "cloud-build@${var.project_id}.iam.gserviceaccount.com"
impersonate_service_account = "cloud-build@${var.project_id}.iam.gserviceaccount.com"
temporary_key_pair_type = "ed25519"
ssh_username = "auto-replaced"
use_os_login = false
}
```
### Operating system and Environment details
- Running in Google Cloud Build. I have tried both the default worker space and private worker pools.
- VM image is a lightly modified version of Debian 11.3. The firewall allows SSH (port 22) and works fine when an external IP is attached, so this is also not the issue.
```yml
source:
repoSource:
projectId: projects/redacted
repoName: redacted
branchName: base-deb11
dir: base-deb11
steps:
- name: 'hashicorp/packer:light'
args:
- '-c'
- packer init base-deb11.gcloud.pkr.hcl
dir: /workspace/base-deb11
entrypoint: bash
- name: 'hashicorp/packer:light'
env:
- PACKER_LOG=1
- PACKER_LOG_PATH=/workspace/packer.log
args:
- '-c'
- packer build base-deb11.gcloud.pkr.hcl
dir: /workspace/base-deb11
entrypoint: bash
```
### Log Fragments and crash.log files
These are the relevant lines from packer.log, with the entire gist still being [linked](https://gist.github.com/oinkbark/9bc2c7a13fb50fe9965c73c6d01a16cd):
```
2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 Finding an available TCP port for IAP proxy
2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 Found available port: 8290 on IP: 0.0.0.0
2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 Setting up proxy to listen on localhost at 8290
2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 Creating tunnel launch script with args []string{"compute", "start-iap-tunnel", "base-deb11", "22", "--local-host-port=localhost:8290", "--zone", "redacted", "--project", "redacted", "--impersonate-service-account='cloud-build@redacted.iam.gserviceaccount.com'"}
2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 [INFO] (google): Prepending inline gcloud setup script with #!/bin/sh
2022/06/30 00:04:09 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:09 stderr: /tmp/gcloud-setup3357350656: line 2: gcloud: not found
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 No error detected after tunnel launch; continuing...
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 Using host value: localhost
2022/06/30 00:04:38 ui: [1;32m==> googlecompute.base-deb11: Using SSH communicator to connect: localhost[0m
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 [INFO] Waiting for SSH, up to timeout: 5m0s
2022/06/30 00:04:38 ui: [1;32m==> googlecompute.base-deb11: Waiting for SSH to become available...[0m
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 Using host value: localhost
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 127.0.0.1:8290: connect: connection refused
```
### Potential Workarounds
Internally, it appears that this plugin is using this command, which is causing the issue:
```
gcloud compute start-iap-tunnel 22 --local-host-port=localhost: --zone --project --impersonate-service-account=''
```
I have been able to get the build to work using these two alternatives:
1. Using tunnel-though
```
gcloud compute ssh --tunnel-through-iap --project --zone
```
2. Using private worker pools. This replaces IAP (ie you disable it) and makes the cloud build node apart of a private worker pool that directly has access to the local IPs of the machines that it creates. This requires some extra firewall rules, IAM permissions, peering, and private service connections.
Contributor guide
Assessment
This issue has not been assessed yet.