[BUG] Add Support for cloud-init/waagent provisioning on "Fedora 38" (RDMA)

Cannot Get '/dev/sr0/ovf-env.xml' suport on boot for Fedora 38 vm
============================================

Impossible to make generalized image: "Boot wait for missing provisioning CD"

Cannot get provisioning data on boot:

_**udf filesystem supported, but cd rom was not present**_

Distro and WALinuxAgent details:
======================

- Distro and Version:

hostnamectl
--------------

```
Static hostname: datalab-base
Icon name: computer-vm
Chassis: vm 🖴
Machine ID: cd9397d4f87d4547b049d115201420ab
Boot ID: 225c7836ccd94a20a07b0deeb5d367bc
Virtualization: microsoft
Operating System: Fedora Linux 38 (Thirty Eight)
CPE OS Name: cpe:/o:fedoraproject:fedora:38
OS Support End: Tue 2024-05-14
OS Support Remaining: 9month 2d
Kernel: Linux 6.4.7-200.fc38.x86_64
Architecture: x86-64
Hardware Vendor: Microsoft Corporation
Hardware Model: Virtual Machine
Firmware Version: Hyper-V UEFI Release v4.1
Firmware Date: Mon 2022-05-09
```

uname -a
-----------

```
Linux datalab-base 6.4.7-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jul 27 20:01:18 UTC 2023 x86_64 GNU/Linux
```

inxi -F
-------

```
System:
Host: datalab-base Kernel: 6.4.7-200.fc38.x86_64 arch: x86_64 bits: 64 Console: pty pts/0
Distro: Fedora release 38 (Thirty Eight)
Machine:
Type: Desktop Mobo: Microsoft model: Virtual Machine v: Hyper-V UEFI Release v4.1
serial: 0000-0011-7863-4539-4444-6382-08 UEFI: Microsoft v: Hyper-V UEFI Release v4.1
date: 05/09/2022
CPU:
Info: dual core model: Intel Xeon Platinum 8370C bits: 64 type: MT MCP cache: L2: 2.5 MiB
Speed (MHz): avg: 1789 min/max: 800/2800 cores: 1: 800 2: 800 3: 2760 4: 2796
Graphics:
Message: No PCI device data found.
Display: server: X.org v: 1.20.14 driver: N/A tty: 168x28
API: OpenGL Message: GL data unavailable in console for root.
Audio:
Message: No device data found.
Network:
Device-1: Mellanox MT27800 Family [ConnectX-5 Virtual Function] driver: mlx5_core
IF: eth1 state: up speed: 100000 Mbps duplex: full mac: 00:0d:3a:2c:2f:95
IF-ID-1: eth0 state: up speed: 100000 Mbps duplex: full mac: 00:0d:3a:2c:2f:95
Drives:
Local Storage: total: 155 GiB used: 3.75 GiB (2.4%)
ID-1: /dev/sda model: Virtual Disk size: 5 GiB
ID-2: /dev/sdb model: Virtual Disk size: 150 GiB
Partition:
ID-1: / size: 4.17 GiB used: 3.49 GiB (83.9%) fs: ext4 dev: /dev/sda4
ID-2: /boot size: 458.3 MiB used: 250 MiB (54.6%) fs: ext4 dev: /dev/sda3
ID-3: /boot/efi size: 199.8 MiB used: 7 MiB (3.5%) fs: vfat dev: /dev/sda2
Swap:
ID-1: swap-1 type: zram size: 8 GiB used: 0 KiB (0.0%) dev: /dev/zram0
Sensors:
Src: lm-sensors+/sys Message: No sensor data found using /sys/class/hwmon or lm-sensors.
Info:
Processes: 155 Uptime: 20m Memory: available: 15.35 GiB used: 673.2 MiB (4.3%) Init: systemd
target: multi-user (3) Shell: Bash inxi: 3.3.27
```

waagent:
----------

```
WALinuxAgent-2.9.0.4 running on fedora 38
Python: 3.11.4
Goal state agent: 2.9.1.1
```

Additional context
---------------------

VM build with [Red Hat Enterprise Linux 8 Image Builder](https://developers.redhat.com/blog/2019/05/08/red-hat-enterprise-linux-8-image-builder-building-custom-system-images)

Log file attached
--------------------
- boot log:

---

```
2023-08-11T13:07:32.631277Z INFO Daemon Daemon Azure Linux Agent Version: 2.9.0.4
2023-08-11T13:07:32.682440Z INFO Daemon Daemon OS: fedora 38
2023-08-11T13:07:32.682987Z INFO Daemon Daemon Python: 3.11.4
2023-08-11T13:07:32.683633Z INFO Daemon Daemon Run daemon
2023-08-11T13:07:32.684112Z INFO Daemon Daemon sr0/ovf.-eml.xmkl
2023-08-11T13:07:32.690127Z INFO Daemon Daemon Unable to get cloud-init enabled status from systemctl: Command '['systemctl', 'is-enabled', 'cloud-init-local.service']' returned non-zero exit status 4.
2023-08-11T13:07:32.946482Z INFO Daemon Daemon Unable to get cloud-init enabled status from service: Command '['service', 'cloud-init', 'status']' returned non-zero exit status 4.
2023-08-11T13:07:32.946795Z INFO Daemon Daemon cloud-init is enabled: False
2023-08-11T13:07:32.947868Z INFO Daemon Daemon Using waagent for provisioning
2023-08-11T13:07:32.949082Z INFO Daemon Daemon Activate resource disk
2023-08-11T13:07:32.949405Z INFO Daemon Daemon Searching gen1 prefix 00000000-0001 or gen2 f8b3781a-1e82-4818-a1c3-63d806ec15bb
2023-08-11T13:07:32.950658Z INFO Daemon Daemon Found device: sdb
2023-08-11T13:07:32.956180Z INFO Daemon Daemon Examining partition table
2023-08-11T13:07:33.194217Z INFO Daemon Daemon GPT not detected, determining filesystem

Fedora Linux 38 (Thirty Eight)
Kernel 6.4.7-200.fc38.x86_64 on an x86_64 (ttyS0)

datalab-base login: 2023-08-11T13:07:33.326590Z INFO Daemon Daemon sfdisk --part-type -f /dev/sdb 1 -n succeeded
2023-08-11T13:07:33.326880Z INFO Daemon Daemon The partition is formatted with ntfs, updating partition type to 83
[ 28.844094] sdb: sdb1
[ 28.864279] sdb: sdb1
2023-08-11T13:07:34.097025Z INFO Daemon Daemon sfdisk --part-type /dev/sdb 1 83 succeeded
[ 28.884385] sdb: sdb1
2023-08-11T13:07:34.106803Z INFO Daemon Daemon Format partition [mkfs.ext4 -F /dev/sdb1]
2023-08-11T13:07:36.551743Z INFO Daemon Daemon Mount resource disk [mount -t ext4 /dev/sdb1 /mnt/resource]
[ 31.335410] EXT4-fs (sdb1): mounted filesystem 99936882-262f-42b4-b918-64a93fea194d r/w with ordered data mode. Quota mode: none.
2023-08-11T13:07:36.560785Z INFO Daemon Daemon Resource disk /dev/sdb is mounted at /mnt/resource with ext4
2023-08-11T13:07:36.565156Z INFO Daemon Daemon Clean protocol and wireserver endpoint
2023-08-11T13:07:36.568526Z INFO Daemon Daemon Running default provisioning handler
2023-08-11T13:07:36.573295Z INFO Daemon Daemon Unable to get cloud-init enabled status from systemctl: Command '['systemctl', 'is-enabled', 'cloud-init-local.service']' returned non-zero exit status 4.
2023-08-11T13:07:36.583481Z INFO Daemon Daemon Unable to get cloud-init enabled status from service: Command '['service', 'cloud-init', 'status']' returned non-zero exit status 4.
2023-08-11T13:07:36.583650Z INFO Daemon Daemon cloud-init is enabled: False
2023-08-11T13:07:36.584698Z INFO Daemon Daemon Copying ovf-env.xml
2023-08-11T13:07:36.585599Z ERROR Daemon Daemon Provisioning failed: [ProtocolError] [CopyOvfEnv] Error mounting dvd: [OSUtilError] Failed to get dvd device from /dev
Inner error: The following devices were found, but none matched the pattern [(sr[0-9]|hd[c-z]|cdrom[0-9]|cd[0-9]|vd[b-z])]: ['sdb1', 'vcsa6', 'vcsu6', 'vcs6', 'vcsa5', 'vcsu5', 'vcs5', 'vcsa4', 'vcsu4', 'vcs4', 'vcsa3', 'vcsu3', 'vcs3', 'vcsa2', 'vcsu2', 'vcs2', 'infiniband', 'kvm', 'ptp_hyperv', 'ptp1', 'vmbus', 'rtc', 'snd', 'vhost-vsock', 'vhost-net', 'vhci', 'vfio', 'uinput', 'ppp', 'mqueue', 'hugepages', 'initctl', 'zram0', 'ptp0', 'block', 'disk', 'sda4', 'sda3', 'sda2', 'sda1', 'sda', 'sdb', 'sg1', 'bsg', 'sg0', 'fb0', 'dri', 'char', 'rfkill', 'uhid', 'net', 'loop-control', 'cuse', 'fuse', 'log', 'stderr', 'stdout', 'stdin', 'fd', 'core', 'pts', 'shm', 'btrfs-control', 'cpu_dma_latency', 'mcelog', 'mapper', 'rtc0', 'input', 'usbmon0', 'udmabuf', 'dma_heap', 'nvram', 'hpet', 'ttyS31', 'ttyS30', 'ttyS29', 'ttyS28', 'ttyS27', 'ttyS26', 'ttyS25', 'ttyS24', 'ttyS23', 'ttyS22', 'ttyS21', 'ttyS20', 'ttyS19', 'ttyS18', 'ttyS17', 'ttyS16', 'ttyS15', 'ttyS14', 'ttyS13', 'ttyS12', 'ttyS11', 'ttyS10', 'ttyS9', 'ttyS8', 'ttyS7', 'ttyS6', 'ttyS5', 'ttyS4', 'ttyS3', 'ttyS2', 'ttyS1', 'ttyS0', 'ptmx', 'autofs', 'userfaultfd', 'snapshot', 'cpu', 'hwrng', 'tty63', 'tty62', 'tty61', 'tty60', 'tty59', 'tty58', 'tty57', 'tty56', 'tty55', 'tty54', 'tty53', 'tty52', 'tty51', 'tty50', 'tty49', 'tty48', 'tty47', 'tty46', 'tty45', 'tty44', 'tty43', 'tty42', 'tty41', 'tty40', 'tty39', 'tty38', 'tty37', 'tty36', 'tty35', 'tty34', 'tty33', 'tty32', 'tty31', 'tty30', 'tty29', 'tty28', 'tty27', 'tty26', 'tty25', 'tty24', 'tty23', 'tty22', 'tty21', 'tty20', 'tty19', 'tty18', 'tty17', 'tty16', 'tty15', 'tty14', 'tty13', 'tty12', 'tty11', 'tty10', 'tty9', 'tty8', 'tty7', 'tty6', 'tty5', 'tty4', 'tty3', 'tty2', 'tty1', 'vcsa1', 'vcsu1', 'vcs1', 'vcsa', 'vcsu', 'vcs', 'tty0', 'console', 'tty', 'kmsg', 'urandom', 'random', 'full', 'zero', 'port', 'null', 'mem', 'vga_arbiter']
(30.85s)
2023-08-11T13:07:36.586553Z INFO Daemon Daemon Detect protocol endpoint
2023-08-11T13:07:36.586965Z INFO Daemon Daemon Clean protocol and wireserver endpoint
2023-08-11T13:07:36.587407Z INFO Daemon Daemon WireServer endpoint is not found. Rerun dhcp handler
2023-08-11T13:07:36.587818Z INFO Daemon Daemon Test for route to 168.63.129.16
2023-08-11T13:07:36.588802Z INFO Daemon Daemon Route to 168.63.129.16 exists
2023-08-11T13:07:36.589150Z INFO Daemon Daemon Wire server endpoint:168.63.129.16
2023-08-11T13:07:36.696707Z INFO Daemon Daemon Fabric preferred wire protocol version:2015-04-05
2023-08-11T13:07:36.697156Z INFO Daemon Daemon Wire protocol version:2012-11-30
2023-08-11T13:07:36.697465Z INFO Daemon Daemon Server preferred version:2015-04-05
2023-08-11T13:07:37.199941Z INFO Daemon Daemon Initializing goal state during protocol detection
2023-08-11T13:07:37.200253Z INFO Daemon Daemon Forcing an update of the goal state.
2023-08-11T13:07:37.206514Z INFO Daemon Fetched a new incarnation for the WireServer goal state [incarnation 1]
2023-08-11T13:07:37.280277Z INFO Daemon Daemon HostGAPlugin version: 1.0.8.141
2023-08-11T13:07:37.304330Z INFO Daemon
2023-08-11T13:07:37.304539Z INFO Daemon Fetched new vmSettings [HostGAPlugin correlation ID: beaddaaf-32be-4c3a-942b-e3cb09bd2244 eTag: 6277160325177573448 source: Fabric]
2023-08-11T13:07:37.305702Z INFO Daemon The vmSettings originated via Fabric; will ignore them.
2023-08-11T13:07:37.306819Z INFO Daemon
2023-08-11T13:07:37.307030Z INFO Daemon Fetching full goal state from the WireServer [incarnation 1]
2023-08-11T13:07:37.473731Z INFO Daemon Downloaded certificate {'thumbprint': '5A2D239A3E1B6DE480E3D1D4A6B6CE717DAD82AB', 'hasPrivateKey': True}
2023-08-11T13:07:37.474357Z INFO Daemon Fetch goal state completed
2023-08-11T13:07:37.478948Z ERROR Daemon Daemon Event: name=WALinuxAgent, op=Provision, message=Provisioning failed: [ProtocolError] [CopyOvfEnv] Error mounting dvd: [OSUtilError] Failed to get dvd device from /dev
Inner error: The following devices were found, but none matched the pattern [(sr[0-9]|hd[c-z]|cdrom[0-9]|cd[0-9]|vd[b-z])]: ['sdb1', 'vcsa6', 'vcsu6', 'vcs6', 'vcsa5', 'vcsu5', 'vcs5', 'vcsa4', 'vcsu4', 'vcs4', 'vcsa3', 'vcsu3', 'vcs3', 'vcsa2', 'vcsu2', 'vcs2', 'infiniband', 'kvm', 'ptp_hyperv', 'ptp1', 'vmbus', 'rtc', 'snd', 'vhost-vsock', 'vhost-net', 'vhci', 'vfio', 'uinput', 'ppp', 'mqueue', 'hugepages', 'initctl', 'zram0', 'ptp0', 'block', 'disk', 'sda4', 'sda3', 'sda2', 'sda1', 'sda', 'sdb', 'sg1', 'bsg', 'sg0', 'fb0', 'dri', 'char', 'rfkill', 'uhid', 'net', 'loop-control', 'cuse', 'fuse', 'log', 'stderr', 'stdout', 'stdin', 'fd', 'core', 'pts', 'shm', 'btrfs-control', 'cpu_dma_latency', 'mcelog', 'mapper', 'rtc0', 'input', 'usbmon0', 'udmabuf', 'dma_heap', 'nvram', 'hpet', 'ttyS31', 'ttyS30', 'ttyS29', 'ttyS28', 'ttyS27', 'ttyS26', 'ttyS25', 'ttyS24', 'ttyS23', 'ttyS22', 'ttyS21', 'ttyS20', 'ttyS19', 'ttyS18', 'ttyS17', 'ttyS16', 'ttyS15', 'ttyS14', 'ttyS13', 'ttyS12', 'ttyS11', 'ttyS10', 'ttyS9', 'ttyS8', 'ttyS7', 'ttyS6', 'ttyS5', 'ttyS4', 'ttyS3', 'ttyS2', 'ttyS1', 'ttyS0', 'ptmx', 'autofs', 'userfaultfd', 'snapshot', 'cpu', 'hwrng', 'tty63', 'tty62', 'tty61', 'tty60', 'tty59', 'tty58', 'tty57', 'tty56', 'tty55', 'tty54', 'tty53', 'tty52', 'tty51', 'tty50', 'tty49', 'tty48', 'tty47', 'tty46', 'tty45', 'tty44', 'tty43', 'tty42', 'tty41', 'tty40', 'tty39', 'tty38', 'tty37', 'tty36', 'tty35', 'tty34', 'tty33', 'tty32', 'tty31', 'tty30', 'tty29', 'tty28', 'tty27', 'tty26', 'tty25', 'tty24', 'tty23', 'tty22', 'tty21', 'tty20', 'tty19', 'tty18', 'tty17', 'tty16', 'tty15', 'tty14', 'tty13', 'tty12', 'tty11', 'tty10', 'tty9', 'tty8', 'tty7', 'tty6', 'tty5', 'tty4', 'tty3', 'tty2', 'tty1', 'vcsa1', 'vcsu1', 'vcs1', 'vcsa', 'vcsu', 'vcs', 'tty0', 'console', 'tty', 'kmsg', 'urandom', 'random', 'full', 'zero', 'port', 'null', 'mem', 'vga_arbiter']
(30.85s), duration=0
2023-08-11T13:07:37.547758Z INFO Daemon Daemon RDMA capabilities are not enabled, skipping
2023-08-11T13:07:37.547965Z INFO Daemon Daemon End of log to /dev/console. The agent will now check for updates and then will process extensions.
2023-08-11T13:07:37.988992Z INFO ExtHandler ExtHandler Azure Linux Agent (Goal State Agent version 2.9.1.1)
2023-08-11T13:07:37.989265Z INFO ExtHandler ExtHandler OS: fedora 38
2023-08-11T13:07:37.989800Z INFO ExtHandler ExtHandler Python: 3.11.4
2023-08-11T13:07:38.018552Z INFO ExtHandler ExtHandler Distro: fedora-38; OSUtil: FedoraOSUtil; AgentService: waagent; Python: 3.11.4; systemd: True; LISDrivers: Absent; logrotate: logrotate 3.21.0;
2023-08-11T13:07:38.018811Z INFO ExtHandler ExtHandler WireServer endpoint 168.63.129.16 read from file
2023-08-11T13:07:38.019707Z INFO ExtHandler ExtHandler Wire server endpoint:168.63.129.16
2023-08-11T13:07:38.032888Z INFO ExtHandler Fetched a new incarnation for the WireServer goal state [incarnation 1]
2023-08-11T13:07:38.037324Z INFO ExtHandler ExtHandler HostGAPlugin version: 1.0.8.141
2023-08-11T13:07:38.037704Z INFO ExtHandler
2023-08-11T13:07:38.037902Z INFO ExtHandler Fetched new vmSettings [HostGAPlugin correlation ID: 469c2960-54cd-413a-80c1-8c8d522d01cc eTag: 6277160325177573448 source: Fabric]
2023-08-11T13:07:38.038924Z INFO ExtHandler The vmSettings originated via Fabric; will ignore them.
2023-08-11T13:07:38.040157Z INFO ExtHandler
2023-08-11T13:07:38.040550Z INFO ExtHandler Fetching full goal state from the WireServer [incarnation 1]
2023-08-11T13:07:38.044136Z INFO ExtHandler ExtHandler Downloading artifacts profile blob
2023-08-11T13:07:38.148669Z INFO ExtHandler Downloaded certificate {'thumbprint': '5A2D239A3E1B6DE480E3D1D4A6B6CE717DAD82AB', 'hasPrivateKey': True}
2023-08-11T13:07:38.149345Z INFO ExtHandler Fetch goal state completed
2023-08-11T13:07:38.162285Z INFO ExtHandler ExtHandler WALinuxAgent-2.9.1.1 running as process 1058
2023-08-11T13:07:38.162485Z INFO ExtHandler ExtHandler ******** OS.EnableFirewallPeriod changed from its default: 300. New value: 30. ********
2023-08-11T13:07:38.164181Z INFO ExtHandler ExtHandler Cgroup monitoring is not supported on ['fedora', '38', 'Thirty Eight', 'Fedora Linux']
2023-08-11T13:07:38.283160Z INFO ExtHandler ExtHandler Starting setup for Persistent firewall rules
2023-08-11T13:07:39.450495Z INFO ExtHandler ExtHandler Firewalld service not running/unavailable, trying to set up waagent-network-setup.service
2023-08-11T13:07:39.450886Z INFO ExtHandler ExtHandler Successfully updated the Binary file /var/lib/waagent/waagent-network-setup.py for firewall setup
2023-08-11T13:07:39.457293Z INFO ExtHandler ExtHandler Unit file version matches with expected version: 1.3, not overwriting unit file
2023-08-11T13:07:39.457689Z INFO ExtHandler ExtHandler Service: waagent-network-setup.service already enabled. No change needed.
2023-08-11T13:07:39.721997Z INFO ExtHandler ExtHandler Logs from the waagent-network-setup.service since system boot:
Aug 11 13:07:28 datalab-base python3[685]: Setting up firewall for the WALinux Agent with args: {'dst_ip': '168.63.129.16', 'uid': '0', 'wait': '-w'}
Aug 11 13:07:28 datalab-base python3[685]: Successfully set the firewall rules
Aug 11 13:07:28 datalab-base systemd[1]: Finished waagent-network-setup.service - Setup network rules for WALinuxAgent.

```
References;
-------------
* https://learn.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-centos#centos-70
* https://learn.microsoft.com/en-us/azure/virtual-machines/linux/redhat-create-upload-vhd

See also:
----------

* https://github.com/Azure/WALinuxAgent/issues/1198
* #1198
* https://learn.microsoft.com/en-us/answers/questions/326296/info-daemon-waiting-for-cloud-init-to-copy-ovf-env
* https://azuretestingsite.com/2022/08/08/migrating-a-centos-virtual-application-to-azure/

```[tasklist]
### Tasks
```