cc_grub_dpkg updates grub-pc or grub-efi debconf keys, but both can become incorrect on BIOS-booted Azure Ubuntu

This bug was originally filed in Launchpad as [LP: #2013419](https://bugs.launchpad.net/cloud-init/+bug/2013419)

Launchpad details

affected_projects = []

assignee = None

assignee_name = None

date_closed = None

date_created = 2023-03-31T01:52:18.234876+00:00

date_fix_committed = 2023-04-05T21:35:36.231101+00:00

date_fix_released = 2023-04-05T21:35:36.231101+00:00

id = 2013419

importance = undecided

is_complete = False

lp_url = https://bugs.launchpad.net/cloud-init/+bug/2013419

milestone = None

owner = r-asiebert

owner_name = Adrien Siebert

private = False

status = incomplete

submitter = r-asiebert

submitter_name = Adrien Siebert

tags = []

duplicates = []

_Launchpad user **Adrien Siebert(r-asiebert)** wrote on 2023-03-31T01:52:18.234876+00:00_

Platform: Azure (generation 1 VMs)
Image used: Ubuntu Server, SKU 20_04-lts (gen 1)
(cloudinit 22.4.2-0ubuntu0~20.04.2 - no cloudinit customization)

Azure generation 1 VMs boot in BIOS mode. Ubuntu comes with both BIOS and UEFI support installed, and cloudinit updates some debconf keys presumably to avoid mismatches when boot packages get updated on new machines:
https://github.com/canonical/cloud-init/blob/ubuntu/22.4.2-0ubuntu0_20.04.2/cloudinit/config/cc_grub_dpkg.py#L148-L149

Even when booted in BIOS mode, updating EFI packages (e.g. grub-efi-amd64-signed or shim-signed) will cause the debconf `grub-efi/install_devices` keys to be updated.
If a discrepancy occurs on the disk ID where GRUB is installed (one scenario below), cloudinit only updates the `grub-pc` debconf keys (link above). The mismatched `grub-efi` key can cause further EFI package upgrades to fail, requiring a user with a shell to validate a prompt for dpkg configuration.

[scenario]
Sample scenario where we encountered this issue, using Packer to build a custom VM image:
* Packer creates a BIOS VM from the base Ubuntu 20.04 image (gen 1).
* cloudinit updates the `grub-pc` key:
```
2023-02-26 08:40:19,507 - cc_grub_dpkg.py[DEBUG]: Setting grub debconf-set-selections with '/dev/disk/by-id/scsi-14d534654202020204f19e7ec574d624f9e27ff405f501bc0','false'
```
* Packages get upgraded. Upgrades to EFI packages results in "Installing grub to /boot/efi" (dpkg logs) and debconf `grub-efi/install_devices` keys to be set, pointing at the Packer VM disk.
* Customized VM gets saved by Packer as an image.

...

* Later, we spin up gen 1 (BIOS) VMs from that image. Its root disk has its own serial ID.
(GRUB partition = scsi-14d53465420202020da118904a05ed740b387a530ae506ac2-part15)
* cloudinit updates the `grub-pc` key:
```
2023-03-07 00:25:44,780 - cc_grub_dpkg.py[DEBUG]: Setting grub debconf-set-selections with '/dev/disk/by-id/scsi-14d534654202020200e6290db5a56ef43ba1f16eef596d653','false'
```
* Later, a headless `apt upgrade` breaks:
```
Setting up shim-signed (1.40.9+15.7-0ubuntu1) ...
mount: /var/lib/grub/esp: special device /dev/disk/by-id/scsi-14d534654202020204f19e7ec574d624f9e27ff405f501bc0-part15 does not exist.
```
```
# debconf-show grub-pc | egrep "grub-efi/install_devices:|grub-pc/install_devices:"
* grub-efi/install_devices: /dev/disk/by-id/scsi-14d534654202020204f19e7ec574d624f9e27ff405f501bc0-part15
* grub-pc/install_devices: /dev/disk/by-id/scsi-14d534654202020200e6290db5a56ef43ba1f16eef596d653
```
[/scenario]

In this situation, when running `apt upgrade` updating an EFI package (or `dpkg --configure -a` once broken) in a shell, a user can manually validate this prompt:
```
┌───────────────────────────────────┤ Configuring shim-signed ├────────────────────────────────────┐
│ The GRUB boot loader was previously installed to a disk that is no longer present, or whose |
| unique identifier has changed for some reason. It is important to make sure that the installed |
| GRUB core image stays in sync with GRUB modules and grub.cfg. Please check again to make sure |
| that GRUB is written to the appropriate boot devices. |
│ |
│ GRUB install devices: |
│ |
│ [*] /dev/sda15 (111 MB; /boot/efi) on 32213 MB Virtual_Disk |
│ |
│ │
│ │
│ │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘
```
Accepting this prompt appears to update/fix the `grub-efi` debconf key.
In my testing, `DEBIAN_FRONTEND=noninteractive` disables the prompt but it instead blows up with the aforementioned `mount: /var/lib/grub/esp...` error, which may be related to https://bugs.launchpad.net/ubuntu/+source/shim-signed/+bug/1940723 --- a bug suspiciously related to Azure VMs.

Reminder: this is all on BIOS-booted VMs, as far as I know UEFI boot is never involved here.

This bug is a follow-up to a quick discussion on https://github.com/canonical/cloud-init/commit/2fd24cc8cb2e2d1b0e00eb8c66573722523a91e7
Support for EFI-booted machines to update grub debconf was introduced in that recent change, although based on the boot mode: if EFI-booted, update debconf `grub-efi`, otherwise update `grub-pc`.

This unfortunately doesn't solve the case above, where BIOS machines have EFI configured and an intermediate/customized image is used.

My uneducated guess is that we may want cloudinit to update either/both debconf keys if BIOS and/or EFI support is *installed*, instead of checking the current boot mode (= presence of `/sys/firmware/efi`).
I do not know how to detect this. Presence of a grub-efi* package? Presence of /boot/efi?

Start with cloudinit/config/cc_grub_dpkg.py around lines 148-149 and the change in commit 2fd24cc8cb2e2d1b0e00eb8c66573722523a91e7. Compare the current /sys/firmware/efi boot-mode check with the installed grub and EFI state described in the Azure BIOS scenario. Done means the agreed behavior is covered for customized BIOS images and headless package upgrades no longer retain an invalid grub-efi device.

Written by the indexing model from the issue text.