grubcmdline notifies reboot handler unnecessarily

Whilst configuring machines with the grubcmdline the check for the command line is not idempotent, so the same machine will reboot every time.

I suspect this is either

• related to crashkernel appearing multiple times in the args (which we should fix in the caller, but should the role does not guard against) or
• resume=UUID gets packed into the cmdline multiple times

The role should either throw a fatal: message, or not notify the reboot handler in these cases

Logs

TASK [stackhpc.linux.grubcmdline : Notify reboot handler] **********************
Friday 20 December 2024  11:22:41 +0000 (0:00:00.049)       0:00:08.921 ******* 
changed: [hv.example.com] => 
  msg: Old command line was BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-503.15.1.el9_5.x86_64 root=UUID=6caabe9f-f426-4105-a38e-52e98545a68a ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=
c38017eb-0531-46fa-bf97-a9265a9caed4 rhgb quiet crashkernel=auto rhgb quiet crashkernel=auto
....
TASK [stackhpc.linux.grubcmdline : Display GRUB_CMDLINE_LINUX_DEFAULT] *********
Friday 20 December 2024  11:22:42 +0000 (0:00:00.039)       0:00:09.543 ******* 
ok: [hv.example.com] => 
  grub_cmdline_linux_default: crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=c38017eb-0531-46fa-bf97-a9265a9caed4 rhgb quiet crashkernel=auto rhgb quiet crashkernel=auto
...
TASK [stackhpc.linux.grubcmdline : Display GRUB_CMDLINE_LINUX] *****************
Friday 20 December 2024  11:22:42 +0000 (0:00:00.038)       0:00:09.582 ******* 
ok: [hv.example.com] => 
  grub_cmdline_linux: crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=c38017eb-0531-46fa-bf97-a9265a9caed4 rhgb quiet crashkernel=auto rhgb quiet crashkernel=auto
.....
.....
  
  
TASK [stackhpc.linux.grubcmdline : Display newly computed GRUB_CMDLINE_LINUX_DEFAULT] ***
Friday 20 December 2024  11:22:42 +0000 (0:00:00.046)       0:00:09.816 ******* 
ok: [hv.example.com] => 
  grub_cmdline_linux_new:
  - crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
  - resume=UUID=c38017eb-0531-46fa-bf97-a9265a9caed4
  - rhgb
  - quiet
  - crashkernel=auto
  - rhgb
  - quiet
  - crashkernel=auto

After reboot:

cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-503.15.1.el9_5.x86_64 root=UUID=6caabe9f-f426-4105-a38e-52e98545a68a ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=c38017eb-0531-46fa-bf97-a9265a9caed4 rhgb quiet crashkernel=auto rhgb quiet crashkernel=auto

Calling Module

  vars:
    kernel_cmdline_args:
      - "rhgb"
      - "quiet"
      - "crashkernel=auto"
    kernel_cmdline_args_remove:
      - hugepage
      - selinux
  roles:
    - name: stackhpc.linux.grubcmdline
      vars:
        kernel_cmdline: "{{ kernel_cmdline_args }}"
        kernel_cmdline_remove: "{{ kernel_cmdline_args_remove }}"

Own thoughts:

• Changing the default parameters on /etc/grub2.cfg or grubby args is working around the existing tooling built into the OS and grub:

Adding args should be done through templating files into /etc/grub.d for each override (RH Grubby Docs and StackOverflow example), such as /etc/grub.d/60-stackhpc-grubcmdline-quiet then simply invoke grubby or grub-mkconfig without args

Removing entries would still edit /etc/default/grub, but ansible.builtin.lineinfile should do the "surgical cuts" instead to minimise the changes to the file rather than trying generate a "new" cmdline using Jinja templates (which could generate wrong and result in an unbootable machine).
Alternatively, a grub script can be templated out /etc/grub.d which iterates over the GRUB_CMDLINE_LINUX* variables and appends every argument back except the matching one using the for in and if statements built into grub script. This would be more correct, but I don't have an adaptable example to hand.

The biggest advantages are:

• Prevents the role fighting upstream changes if they change the default cmdline (e.g. in-place OS upgrades)
• Would avoid the problems with the UUID
• Makes troubleshooting easy to diagnose if it's a distro or kernel flag change: mv /etc/grub.d/*stackhpc-grubcmdline /etc/grub.d.bak && grubby ...
• Idempotent by default, template through ansible then use a handler for re-running grubby and notify reboot if any template / lineinfile has changed

If not the above changes:

• I think old_cmdline != kernel_cmdline | select() | sort | list is potentially brittle as there will be more problems like this in the future. Instead it could be something akin to the following pseudo code:

changed_when: not(all(grub_cmdline_linux_new in kernel_cmdline)) or any(grub_cmdline_linux_remove in kernel_cmdline)

This would only check for keywords the user has explicitly set in the role, rather than being affected by parameters (such as root=UUID=abc...) which they have not specified at all. However, this still leaves the potential problems around directly generating /etc/default/grub

Start with the stackhpc.linux.grubcmdline role entry point and the kernel_cmdline and kernel_cmdline_remove variables shown in the calling module. Reproduce the duplicated crashkernel, rhgb, quiet, and resume arguments, then trace when the reboot handler is notified. Done means repeated configuration is idempotent and does not trigger an unnecessary reboot, or fails with the specified fatal message.