cloud-init: write_files: does not run on instance creation (init runs twice in logs?)

Hello Team,

We have a very simple Package template and then some terraform that creates VM resources using this template. We have noticed some strange behaviour when it comes to using Cloud-init. Specifically:

• write_files: doesn't work correctly at all on first instance creation.
• init seems to (_at least according to the cloud-init.log) run twice?

Our package template is nothing fancy and mostly just consists of:

{
  "variables": {
    "iso_url": "https://packages.vmware.com/photon/3.0/Rev3/iso/photon-minimal-3.0-a383732.iso",
    "iso_checksum": "c2883a42e402a2330d9c39b4d1e071cf9b3b5898",
    "vm-cpu-num": "1",
    "vm-disk-size": "25600",
    "vm-mem-size": "1024",
    "vm-name": "packer-devops-photon-3",
    "vm_notes": "Photon OS 3.x - {{isotime \"2006-01-02\"}}",
    "vsphere-cluster": "",
    "vsphere-datacenter": "",
    "vsphere-datastore": "",
    "vsphere-network": "",
    "vsphere-server": "",
    "vsphere-host": "",
    "vsphere-user": "{{ env `VSPHERE_USERNAME` }}",
    "vsphere-password": "{{ env `VSPHERE_PASSWORD` }}"
  },
  "builders": [
    {
      "CPUs": 1,
      "RAM": 1024,
      "RAM_reserve_all": false,
      "boot_command": [
        "<esc><wait>",
        "vmlinuz initrd=initrd.img root=/dev/ram0 loglevel=3 ks=http://x.x.x.x/repository/ks/kickstart.json photon.media=cdrom",
        "<enter>"
      ],
      "shutdown_command": "shutdown -h now",
      "cluster": "{{user `vsphere-cluster`}}",
      "convert_to_template": true,
      "datacenter": "{{user `vsphere-datacenter`}}",
      "datastore": "{{user `vsphere-datastore`}}",
      "host": "{{user `vsphere-host`}}",
      "iso_target_path": "[xxx] packer_cache/",
      "disk_controller_type": "pvscsi",
      "guest_os_type": "vmwarePhoton64Guest",
      "iso_url": "{{user `iso_url`}}",
      "iso_checksum": "{{user `iso_checksum`}}",
      "network_adapters": [
        {
          "network": "{{user `vsphere-network`}}",
          "network_card": "vmxnet3"
        }
      ],
      "notes": "{{user `vm_notes`}}",
      "password": "{{user `vsphere-password`}}",
      "ssh_username": "root",
      "ssh_password": "***",
      "storage": [
        {
          "disk_size": 25000,
          "disk_thin_provisioned": true
        }
      ],
      "type": "vsphere-iso",
      "username": "{{user `vsphere-user`}}",
      "vcenter_server": "{{user `vsphere-server`}}",
      "vm_name": "{{user `vm-name`}}",
      "folder": "aom/templates"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "scripts": [
        "./scripts/update.sh",
        "./scripts/setup.sh",
        "./scripts/cleanup.sh"
      ]
    },
    {
      "type": "shell",
      "inline": "echo 'Template build complete ({{isotime \"2006-01-02 03:04:05\"}})!'"
    }
  ]
}

The provisioning scripts don't do much either:

./scripts/update.sh:

#!/bin/sh

set -e

# Update
# tdnf -y update

./scripts/setup.sh:

#!/bin/sh

set -e

systemctl start docker
systemctl enable docker
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

And finally:

./scripts/cleanup.sh:

#!/bin/sh

set -ex

# Clean up TDNF cache
tdnf clean all

# Clean up logs
cat /dev/null > /var/log/wtmp 2> /dev/null
#logrotate -f /etc/logrotate.conf 2> /dev/null
find /var/log -type f -delete
rm -rf /var/log/journal/*
rm -f /var/lib/dhcp/*

# Cleanup cloud-init
systemctl stop cloud-init
cloud-init clean

# Cleanup root account
unset HISTFILE && history -c && rm -fr /root/.bash_history

# Cleanup system config
echo > /etc/machine-id
echo > /etc/hostname

# Zero out the free space to save space in the final image, blocking 'til
# written otherwise, the disk image won't be zeroed, and/or Packer will try to
# kill the box while the disk is still full and that's bad.  The dd will run
# 'til failure, so (due to the 'set -e' above), ignore that failure.  Also,
# really make certain that both the zeros and the file removal really sync; the
# extra sleep 1 and sync shouldn't be necessary, but...)
#dd if=/dev/zero of=/EMPTY bs=1M || true
sync
sleep 1
sync
#rm -f /EMPTY
#sync
#sleep 1
#sync

Note that cloud-init clean we run during the cleanup phase. This gets packaged and uploaded to vSphere and we boot from this. But this is where cloud-init falls over.

Attached is the cloud-init.log from one of our instances where we see this behavior.

Once the instance is up and running (with the half-ran cloud-init config) if we shell into it and run cloud-init clean && reboot it correctly runs all of the cloud-init config. But this is not ideal as it doesn't run it correctly in the first place on initial creation/boot.

Thanks!
cloud-init.log

Start with scripts/cleanup.sh and the attached cloud-init.log, focusing on the cloud-init clean step and the stages reported during the initial boot. Compare that behavior with the successful cloud-init clean && reboot run described in the issue. Done means identifying and fixing the first-boot sequencing problem so write_files and the full cloud-init configuration run without manual intervention.