udevadm settle failure on Azure Hirsute

This bug was originally filed in Launchpad as [LP: #1943715](https://bugs.launchpad.net/cloud-init/+bug/1943715)

Launchpad details

affected_projects = []

assignee = None

assignee_name = None

date_closed = None

date_created = 2021-09-15T12:59:41.721809+00:00

date_fix_committed = None

date_fix_released = None

id = 1943715

importance = low

is_complete = False

lp_url = https://bugs.launchpad.net/cloud-init/+bug/1943715

milestone = None

owner = falcojr

owner_name = James Falcon

private = False

status = triaged

submitter = falcojr

submitter_name = James Falcon

tags = []

duplicates = []

_Launchpad user **James Falcon(falcojr)** wrote on 2021-09-15T12:59:41.721809+00:00_

When launching a daily hirsute image, more often than not, we see a failure and traceback in the cloud-init logs:

2021-09-14 16:42:40,580 - cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'device': 'ephemeral0.1'}]
2021-09-14 16:42:40,580 - cc_disk_setup.py[DEBUG]: ephemeral0.1 is mapped to disk=/dev/disk/cloud/azure_resource part=1
2021-09-14 16:42:40,580 - cc_disk_setup.py[DEBUG]: Creating new filesystem.
2021-09-14 16:42:40,581 - subp.py[DEBUG]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=True)
2021-09-14 16:44:40,593 - util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 120.012 seconds
2021-09-14 16:44:40,593 - util.py[WARNING]: Failed during filesystem operation
Unexpected error while running command.
Command: ['udevadm', 'settle']
Exit code: 1
Reason: -
Stdout:
Stderr:
2021-09-14 16:44:40,593 - util.py[DEBUG]: Failed during filesystem operation
Unexpected error while running command.
Command: ['udevadm', 'settle']
Exit code: 1
Reason: -
Stdout:
Stderr:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 157, in handle
util.log_time(logfunc=LOG.debug,
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
ret = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 886, in mkfs
assert_and_settle_device(device)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 765, in assert_and_settle_device
util.udevadm_settle()
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2608, in udevadm_settle
return subp.subp(settle_cmd)
File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 293, in subp
raise ProcessExecutionError(stdout=out, stderr=err,
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['udevadm', 'settle']
Exit code: 1
Reason: -
Stdout:
Stderr:
2021-09-14 16:44:40,636 - handlers.py[DEBUG]: finish: init-network/config-disk_setup: SUCCESS: config-disk_setup ran successfully

We should investigate the root cause.

Start in cloudinit/config/cc_disk_setup.py at assert_and_settle_device() and mkfs(), then follow util.udevadm_settle() in cloudinit/util.py. Reproduce the udevadm settle failure when launching an Azure Hirsute daily image using the logged scenario; done means identifying the root cause and verifying a fix for the filesystem setup traceback.

Written by the indexing model from the issue text.