hashicorp / hashicorp/packer-plugin-googlecompute
Feature addition: zones-list to help deal with STOCKOUT issues in GCP
- Dominant language
- Go
- Stars
- 33
- Forks
- 72
- Avg merge
- 6h 47m
- Merged PRs (30d)
- 3
Description
#### Community Note
Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request.
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
#### Description
When GCE has a capacity stockout in the configured zone, `Instances.Insert` is accepted but the zone operation fails with `ZONE_RESOURCE_POOL_EXHAUSTED` (or `ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS`), and the build dies:
```
==> googlecompute: Error creating instance: The zone 'projects//zones/us-east1-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later.
```
The builder currently takes a single `zone`, so there is no way to express "any zone in this region is fine". Lately we are seeing stockouts routinely persist longer than any reasonable retry window (often hours, regularly half-days), so retrying in place doesn't help; trying the next zone almost always does. Today the only workarounds live outside Packer (wrapper scripts or CI-level retries that rewrite the zone between attempts).
I'd proposing an optional `zones` list: when instance creation fails with a capacity error, the builder tries the next candidate zone before giving up. Existing single-`zone` behavior would be unchanged, and non-capacity errors would still fail immediately.
One small prereq that would need an edit: the operation waiters (`refreshZoneOp` / `refreshGlobalOp` / `refreshRegionOp` in `lib/common/driver_gce.go`) currently flatten `compute.Operation.Error.Errors[]` into plain `errors.New(e.Message)`, discarding the structured `Code` field. Preserving the code in a typed error is what makes stockout classification possible without substring-matching the human-readable message, and it would benefit any future error-specific handling (quota, etc.) independently of this feature.
#### Use Case(s)
Scheduled image-bake pipelines. We run nightly/weekly Packer builds across several image families, and GCE stockouts in a single zone (most recently for newer machine series) fail otherwise-healthy builds several times a month. Any zone in the build region would serve equally well, since the artifact is an image, not a running instance. This seems broadly applicable to anyone running unattended image builds on machine types with tighter capacity (newer generations, GPU shapes, etc.).
#### Potential configuration
```hcl
source "googlecompute" "example" {
project_id = "my-project"
zone = "us-east1-b"
# Optional: additional zones to try, in order, if instance creation
# fails with a capacity error in the previous zone.
zones = ["us-east1-c", "us-east1-d"]
machine_type = "n4-standard-4"
# ...
}
```
Assumptions / judgement calls that might be different from what ya'll might want for something like this
- `zones` as a supplement to `zone` (as shown) vs. a mutually-exclusive alternative to it.
- Whether to restrict candidates to the region derived from `zone`, since `region` is currently inferred from it.
- Which operation error codes count as "capacity" (`ZONE_RESOURCE_POOL_EXHAUSTED`, `ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS`, others?).
#### Potential References
- GCE troubleshooting doc for resource availability errors: https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-vm-creation#resource_availability
- `compute.OperationErrorErrors` carries the machine-readable `Code` at failure time: https://pkg.go.dev/google.golang.org/api/compute/v1#OperationErrorErrors
- An earlier error-code-specific retry handling in this plugin: #34 (treated GCE error 4047 as retryable), #324 (retry loop around `ImportSshPublicKey` on HTTP 409)
- Prior art for "try candidates in order": `source_image_project_id` already accepts a list and falls back through it during image lookup.
---
Happy to do the implementation. My plan would be two PRs: first the typed operation-error change then the `zones` fallback built on it, with mock-driver coverage for the per-zone failure sequencing.
Contributor guide
Research direction
Start with lib/common/driver_gce.go and the refreshZoneOp, refreshGlobalOp, and refreshRegionOp waiters to understand the current operation-error handling. Then trace the googlecompute builder's zone configuration and instance-creation flow. Done should include the agreed zones behavior, typed operation errors, and mock-driver coverage for per-zone failure sequencing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- cloud, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100