sr0 not available at generator timeframe causes cloud-init.target not run

This bug was originally filed in Launchpad as [LP: #1940791](https://bugs.launchpad.net/cloud-init/+bug/1940791)

Launchpad details

affected_projects = ['cloud-images']

assignee = None

assignee_name = None

date_closed = None

date_created = 2021-08-23T03:02:20.779310+00:00

date_fix_committed = None

date_fix_released = None

id = 1940791

importance = undecided

is_complete = False

lp_url = https://bugs.launchpad.net/cloud-init/+bug/1940791

milestone = None

owner = esj

owner_name = Éric St-Jean

private = False

status = triaged

submitter = achasen

submitter_name = Adam Chasen

tags = []

duplicates = [1961832]

_Launchpad user **Adam Chasen(achasen)** wrote on 2021-08-23T03:02:20.779310+00:00_

Focal image cloud-init generator reports:
'cloud-init is enabled but no datasource found, disabling'

looks to be related to ds-identify not finding the cdrom drive (and caching it) on first run. Not sure why /dev/sr0 would not be available early enough.

cat /run/cloud-init/ds-identify.log
...
ISO9660_DEVS=
...
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 1.20s] returning 1
root@ubuntu:~# /usr/lib/cloud-init/ds-identify --force
[up 200.71s] ds-identify --force
...
ISO9660_DEVS=/dev/sr0=cidata
...
Found single datasource: NoCloud
[up 200.79s] returning 0

Booting https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img as of Aug 22, 2021 in KVM (created with virt-install and libvirt) along with cloud-config ISO

$ cat /tmp/cloud
#cloud-config
hostname: proxy1
$ cloud-localds /tmp/test.iso /tmp/cloud

cloud-init.target never reached and network doesn't come up (default behavior for cloud-init is eth0 DHCP). If I manually start `systemctl start cloud-init.target` then I get what I expected, but by then it is "too late" and I also have to kick systemd-networkd.

cloud-init starts up as expected with the same environment when using Bionic (https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img)

The focal image never touches cloud-init.target. Note that there is no reverse dependency in focal.

root@ubuntu:~# systemctl list-dependencies --reverse cloud-init.target
cloud-init.target

Both images have default target of "graphical.target"

There is mention of a "generator" and "detection" in the cloud-init docs. https://cloudinit.readthedocs.io/en/latest/topics/boot.html

The generator appears to be what is adding the "wants" of cloud-init.target to multi-user.target
from /lib/systemd/system-generators/cloud-init-generator:
    local target_name="multi-user.target" gen_d="$early_d"
    local link_path="$gen_d/${target_name}.wants/${CLOUD_TARGET_NAME}"

Bionic:
root@proxy1:~# systemctl get-default
graphical.target
root@proxy1:~#
UNIT LOAD ACTIVE SUB DESCRIPTION
basic.target loaded active active Basic System
cloud-config.target loaded active active Cloud-config availability
cloud-init.target loaded active active Cloud-init target
...
root@proxy1:~# systemctl list-dependencies --reverse cloud-init.target
cloud-init.target
● └─multi-user.target
● └─graphical.target
root@proxy1:/etc/systemd/system# cat /run/cloud-init/cloud-init-generator.log
/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (/proc/cmdline): BOOT_IMAGE=/boot/vmlinuz-4.15.0-154-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=0
ds-identify _RET=found
enabled via /run/systemd/generator.early/multi-user.target.wants/cloud-init.target -> /lib/systemd/system/cloud-init.target

Focal:
root@ubuntu:~# systemctl get-default
graphical.target
root@ubuntu:~# systemctl list-units --type=target --all
  UNIT LOAD ACTIVE SUB >
  basic.target loaded active activ>
  blockdev@dev-disk-by\x2dlabel-cloudimg\x2drootfs.target loaded inactive dead >
  blockdev@dev-disk-by\x2dlabel-UEFI.target loaded inactive dead >
  blockdev@dev-loop0.target loaded inactive dead >
  blockdev@dev-loop1.target loaded inactive dead >
  blockdev@dev-loop2.target loaded inactive dead >
  blockdev@dev-vda15.target loaded inactive dead >
  cloud-config.target loaded inactive dead >
  cloud-init.target loaded inactive dead >
root@ubuntu:~# systemctl list-unit-files
...
cloud-config.service enabled enabled
cloud-final.service enabled enabled
cloud-init-local.service enabled enabled
cloud-init.service enabled enabled
...
root@ubuntu:~# systemctl list-dependencies --reverse cloud-init.target
cloud-init.target
root@ubuntu:~# systemctl list-dependencies cloud-init.target
cloud-init.target
● ├─cloud-config.service
● ├─cloud-final.service
● ├─cloud-init-local.service
● └─cloud-init.service

root@ubuntu:~# cat /run/cloud-init/cloud-init-generator.log
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (/proc/cmdline): BOOT_IMAGE=/boot/vmlinuz-5.4.0-1045-kvm root=PARTUUID=14530a28-f129-4b51-a64e-c64075fae7c7 ro console=tty1 console=ttyS0 panic=-1
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=1
ds-identify _RET=notfound
cloud-init is enabled but no datasource found, disabling
already disabled: no change needed [no /run/systemd/generator.early/multi-user.target.wants/cloud-init.target]

Additional Resources:
Possibly same issue https://bugzilla.redhat.com/show_bug.cgi?id=1820540