hashicorp / hashicorp/packer-plugin-oracle
Packer hangs and ctrl+c does not abort when OCI region is out of capacity.
- Dominant language
- Go
- Stars
- 14
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
#### Overview of the Issue
## Note: See debug log, Out of Capacity is clearly communicated by the OCI SDK, just packer does not handle it.
When an OCI region is out of Capacity, the OCI WebUI will present the following error:
> Out of capacity for shape VM.Standard.E4.Flex in availability domain AD-1. Create the instance in a different availability domain or try again later. If you're using a previous generation shape, please create the instance using a current generation shape instead. For current generation shapes, create the instance in a different availability domain, avoid choosing a fault domain, use a smaller shape or different shape series, or wait a few minutes and try again.

However, packer just hangs, and there are no messages after `Creating instance...`
```bash
$ packer build -only=oracle-oci.jed-amd64 templates/debug/.
oracle-oci.jed-amd64: output will be in this color.
==> oracle-oci.jed-amd64: Creating temporary ssh key for instance...
==> oracle-oci.jed-amd64: Creating instance...
Cancelling build after receiving interrupt
```
Hitting CTRL+C just results in `Cancelling build after receiving interrupt`, but it never actually cleans up.
#### Reproduction Steps
Enable debug logging on both packer and the OCI sdk, if you dont do this for the OCI SDK, the logs are useless.
```bash
export PACKER_LOG=1
export PACKER_LOG_PATH="packer.log"
export OCI_GO_SDK_DEBUG=verbose
```
Deploy to OCI region that has insufficient Capacity, example: any E4 flex size to JED, or large E4 sizes to IAD-AD1 at certain times of day.
### Plugin and Packer version
Packer v1.8.3
### Simplified Packer Buildfile
The only values you need to fix are the compartment_ocid, and the regional public subnet ids within that compartment:
```hcl
variable "compartment_ocid" {
type = string
description = "Compartment ID to build the image"
default = "ocid1.compartment.oc1..aaaaaaaaaaaaaaaa"
}
variables {
regions = {
iad = {
region = "us-ashburn-1"
region_ad = "YOUR_ACCOUNT_AD_PREFIX_HERE:US-ASHBURN-AD-1"
subnet_ocid = "ocid1.subnet.oc1.iad.aaaaaaaaaaaaaaaa"
}
jed = {
region = "me-jeddah-1"
region_ad = "YOUR_ACCOUNT_AD_PREFIX_HERE:ME-JEDDAH-1-AD-1"
subnet_ocid = "ocid1.subnet.oc1.me-jeddah-1.aaaaaaaaaaaaaaaa"
}
}
}
```
Everything else below is as needed to run.
```hcl
// export PACKER_LOG=1
// export PACKER_LOG_PATH="packer.log"
// export OCI_GO_SDK_DEBUG=verbose
// packer build -only=oracle-oci.jed-amd64 templates/debug/.
packer {
required_plugins {
oracle = {
version = ">= 1.0.1"
source = "github.com/hashicorp/oracle"
}
}
}
locals {
datestamp = var.datestamp != null ? var.datestamp : "${formatdate("YYYY.MM.DD", timestamp())}"
image_name_prefix = var.image_name_prefix != null ? var.image_name_prefix : "${regex_replace(var.source_image_name_regex, "[^a-zA-Z0-9-_]+", "")}"
ssh_username = var.ssh_username != "" ? var.ssh_username : length(regexall("ubuntu", lower(var.source_image_name_regex))) > 0 ? "ubuntu" : length(regexall("oracle", lower(var.source_image_name_regex))) > 0 ? "opc" : null
}
build {
dynamic "source" {
for_each = var.regions
labels = ["source.oracle-oci.base-image"]
content {
name = "${source.key}-amd64"
region = source.value.region
availability_domain = source.value.region_ad
subnet_ocid = source.value.subnet_ocid # public subnet, so we can ssh to the public IP
shape = "VM.Standard.E4.Flex"
image_name = "${local.image_name_prefix}-amd64-${local.datestamp}"
instance_name = "packer-${local.image_name_prefix}-amd64-${local.datestamp}"
}
}
provisioner "shell" {
inline = ["uname -a"]
pause_before = "5s"
}
}
source "oracle-oci" "base-image" {
compartment_ocid = var.compartment_ocid
image_compartment_ocid = var.image_compartment_ocid
skip_create_image = var.skip_create_image
ssh_username = local.ssh_username
use_private_ip = "false"
base_image_filter {
display_name_search = var.source_image_name_regex
}
create_vnic_details {
assign_public_ip = "true"
}
shape_config {
ocpus = var.shape_cpus
memory_in_gbs = var.shape_memory
}
instance_tags = {
testing = "yes"
}
tags = {
CreationDate = "${legacy_isotime("20060102 03:04:05 MST")}"
}
}
variable "compartment_ocid" {
type = string
description = "Compartment ID to build the image"
default = "ocid1.compartment.oc1..aaaaaaaaaaaaaaaa"
}
variables {
regions = {
iad = {
region = "us-ashburn-1"
region_ad = "YOUR_ACCOUNT_AD_PREFIX_HERE:US-ASHBURN-AD-1"
subnet_ocid = "ocid1.subnet.oc1.iad.aaaaaaaaaaaaaaaa"
}
jed = {
region = "me-jeddah-1"
region_ad = "YOUR_ACCOUNT_AD_PREFIX_HERE:ME-JEDDAH-1-AD-1"
subnet_ocid = "ocid1.subnet.oc1.me-jeddah-1.aaaaaaaaaaaaaaaa"
}
}
}
variable "datestamp" {
type = string
description = "datestamp for the image name"
default = null
}
variable "ssh_username" {
type = string
description = "SSH username for the image, must already exist on image, opc for Oracle Linux, ubuntu for Ubuntu, etc."
default = ""
}
variable "image_compartment_ocid" {
type = string
description = "Compartment ID to store the built image"
default = null
}
variable "skip_create_image" {
type = bool
description = "Skip creating the image"
default = false
}
variable "image_name_prefix" {
type = string
description = "Image name prefix, ex: Ubuntu-22.04-Minimal"
default = null
}
variable "shape_cpus" {
type = number
description = "Number of OCPUs for the instance"
default = 4
}
variable "shape_memory" {
type = number
description = "Memory in GB for the instance"
default = 8
}
variable "source_image_name_regex" {
type = string
description = "Regex for the image name"
default = "^Canonical-Ubuntu-22.04-Minimal.+"
}
```
### Operating system and Environment details
macOS 12.6 aarch64
### Log Fragments and crash.log files
```log
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Host: iaas.me-jeddah-1.oraclecloud.com
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: User-Agent: Oracle-GoSDK/65.4.0 (darwin/arm64; go/go1.17.13)
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Content-Length: 1030
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Accept: */*
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Authorization: Signature version="1",headers="date (request-target) host content-length content-type x-content-sha256",keyId="ocid1.tenancy.oc1..aaa/ocid1.user.oc1..aaa/aaa",algorithm="rsa-sha256",signature="aaa"
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Content-Type: application/json
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Date: Sat, 08 Oct 2022 21:11:49 GMT
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Opc-Client-Info: Oracle-GoSDK/65.4.0
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Opc-Client-Retries: true
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Opc-Request-Id: e4d053fdb85f96af70d3eb766506bffc
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Opc-Retry-Token: jUqwKoFq4H4RN0hL51w6E2dLsjl42UWo
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: X-Content-Sha256: RXJto/CCYp57nybcGfWTQ8l2TrlEzO7uvpjkUFLFexo=
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Accept-Encoding: gzip
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin:
2022/10/08 15:18:49 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: {"availabilityDomain":"wCio:ME-JEDDAH-1-AD-1","compartmentId":"ocid1.compartment.oc1..aaa","createVnicDetails":{"assignPublicIp":true,"subnetId":"ocid1.subnet.oc1.me-jeddah-1.aaa"},"displayName":"packer-Canonical-Ubuntu-2204-Minimal-amd64-2022.10.08","freeformTags":{"testing":"yes"},"metadata":{"ssh_authorized_keys":"ssh-rsa aaa\n"},"shape":"VM.Standard.E4.Flex","shapeConfig":{"memoryInGBs":8,"ocpus":4},"sourceDetails":{"imageId":"ocid1.image.oc1.me-jeddah-1.aaaaaaaamn2qcvwepf7uwmhkw43nkn5nkebbukm4675q7o3v6wqvvcpwjspa","sourceType":"image"}}
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: INFO 2022/10/08 15:18:51.681121 client.go:411: Dump Response HTTP/1.1 500 Internal Server Error
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Content-Length: 69
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Connection: keep-alive
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Content-Type: application/json
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Date: Sat, 08 Oct 2022 21:11:51 GMT
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: Opc-Request-Id: aaa
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: X-Content-Type-Options: nosniff
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin:
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: {
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: "code" : "InternalError",
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: "message" : "Out of host capacity."
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: }
2022/10/08 15:18:51 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: DEBUG 2022/10/08 15:18:51.681190 asm_arm64.s:1133: waiting 4m16.103s before retrying operation
2022/10/08 15:19:43 ui error: Cancelling build after receiving interrupt
2022/10/08 15:19:43 packer-provisioner-shell plugin: Received interrupt signal (count: 1). Ignoring.
2022/10/08 15:19:43 Cancelling builder after context cancellation context canceled
2022/10/08 15:19:43 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: 2022/10/08 15:09:43 Received interrupt signal (count: 1). Ignoring.
2022/10/08 15:20:47 packer-provisioner-shell plugin: Received interrupt signal (count: 2). Ignoring.
2022/10/08 15:20:47 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: 2022/10/08 15:10:47 Received interrupt signal (count: 2). Ignoring.
2022/10/08 15:20:47 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: 2022/10/08 15:10:47 Received interrupt signal (count: 3). Ignoring.
2022/10/08 15:20:47 packer-provisioner-shell plugin: Received interrupt signal (count: 3). Ignoring.
2022/10/08 15:31:12 packer-plugin-oracle_v1.0.3_x5.0_darwin_arm64 plugin: 2022/10/08 15:10:47 Received interrupt signal (count: 4). Ignoring.
2022/10/08 15:31:12 packer-provisioner-shell plugin: Received interrupt signal (count: 4). Ignoring.
```
Contributor guide
Assessment
This issue has not been assessed yet.