SSM trying to install patches that are not applicable.

I am running serveral hundred RHEL 8.2 EC2 and based on the logs it appears AWS SSM is trying to install patches that is shouldn't then erroring out and not completing patching causing additional issues such a not rebooting.

The /var/log/dnf.log is truncated, but I found another copy of the output under
-rw------- 1 root root 286059 Dec 2 21:13 /var/lib/amazon/ssm/InstanceID/document/orchestration/CMDID/runDocument/InstallNoReboot/InstallNoReboot/awsrunShellScript/PatchLinux/stdout

which has something that caught my eye:
12/02/2023 21:04:22 root [INFO]: Running command ['dnf', 'clean', 'all']
12/02/2023 21:04:24 root [INFO]: Parsing response...
12/02/2023 21:04:24 root [INFO]: Running command ['dnf', 'makecache']
12/02/2023 21:05:02 root [INFO]: Parsing response...
12/02/2023 21:05:02 root [INFO]: Installation process started:

12/02/2023 21:05:02 root [INFO]: Running command ['dnf', 'list', 'installed']
12/02/2023 21:05:06 root [INFO]: Parsing response...
12/02/2023 21:05:06 root [INFO]: Running command ['dnf', 'module', 'list', '--enabled']
12/02/2023 21:05:12 root [INFO]: Parsing response...
12/02/2023 21:05:12 root [INFO]: Running command ['dnf', '--available', 'repoquery', '--queryformat', '%{NAME}.%{ARCH} %{EVR} %{REPONAME} %{BUILDTIME}']
12/02/2023 21:05:18 root [INFO]: Parsing response...
12/02/2023 21:05:50 root [INFO]: Running command ['rpm', '-qa', '--queryformat', '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{INSTALLTIME}\n']
12/02/2023 21:05:51 root [INFO]: Parsing response...
12/02/2023 21:05:53 root [INFO]: Installed count: 379, Installed Other count: 675, Failed count: 0, Not Applicable count: 7766, Missing count: 15, Installed Rejected count: 0
12/02/2023 21:05:53 root [INFO]: Running command ['dnf', 'install', '-y', 'NetworkManager-1:1.22.8-10.el8_2.x86_64']
12/02/2023 21:06:04 root [INFO]: Parsing response...
12/02/2023 21:06:04 root [INFO]: Running command ['dnf', 'install', '-y', 'NetworkManager-libnm-1:1.22.8-10.el8_2.x86_64']
12/02/2023 21:06:09 root [INFO]: Parsing response...
12/02/2023 21:06:09 root [INFO]: Running command ['dnf', 'install', '-y', 'NetworkManager-team-1:1.22.8-10.el8_2.x86_64']
12/02/2023 21:06:14 root [INFO]: Parsing response...
12/02/2023 21:06:14 root [INFO]: Running command ['dnf', 'install', '-y', 'NetworkManager-tui-1:1.22.8-10.el8_2.x86_64']
12/02/2023 21:06:20 root [INFO]: Parsing response...
12/02/2023 21:06:20 root [INFO]: Running command ['dnf', 'install', '-y', 'ansible-0:7.2.0-1.el8.noarch']
12/02/2023 21:06:25 root [INFO]: Parsing response...
12/02/2023 21:06:25 root [WARNING]: Failed to install package: ansible-0:7.2.0-1.el8.noarch
12/02/2023 21:06:25 root [INFO]: Running command ['dnf', 'install', '-y', 'insights-client-0:3.2.3-1.el8_2.noarch']
12/02/2023 21:06:38 root [INFO]: Parsing response...
12/02/2023 21:06:38 root [INFO]: Running command ['dnf', 'install', '-y', 'kernel-0:4.18.0-193.119.1.el8_2.x86_64']
12/02/2023 21:07:50 root [INFO]: Parsing response...
12/02/2023 21:07:50 root [INFO]: Running command ['dnf', 'install', '-y', 'kernel-core-0:4.18.0-193.119.1.el8_2.x86_64']
12/02/2023 21:07:58 root [INFO]: Parsing response...
12/02/2023 21:07:58 root [INFO]: Running command ['dnf', 'install', '-y', 'kernel-headers-0:4.18.0-193.119.1.el8_2.x86_64']
12/02/2023 21:08:20 root [INFO]: Parsing response...
12/02/2023 21:08:20 root [INFO]: Running command ['dnf', 'install', '-y', 'kernel-modules-0:4.18.0-193.119.1.el8_2.x86_64']
12/02/2023 21:08:28 root [INFO]: Parsing response...
12/02/2023 21:08:28 root [INFO]: Running command ['dnf', 'install', '-y', 'kernel-tools-0:4.18.0-193.119.1.el8_2.x86_64']
12/02/2023 21:08:50 root [INFO]: Parsing response...
12/02/2023 21:08:50 root [INFO]: Running command ['dnf', 'install', '-y', 'kernel-tools-libs-0:4.18.0-193.119.1.el8_2.x86_64']
12/02/2023 21:08:58 root [INFO]: Parsing response...
12/02/2023 21:08:58 root [INFO]: Running command ['dnf', 'install', '-y', 'microcode_ctl-4:20191115-4.20231009.1.el8_2.x86_64']
12/02/2023 21:10:26 root [INFO]: Parsing response...
12/02/2023 21:10:26 root [INFO]: Running command ['dnf', 'install', '-y', 'python3-paramiko-0:2.12.0-1.el8.noarch']
12/02/2023 21:10:36 root [INFO]: Parsing response...
12/02/2023 21:10:36 root [WARNING]: Failed to install package: python3-paramiko-0:2.12.0-1.el8.noarch
12/02/2023 21:10:36 root [INFO]: Running command ['dnf', 'install', '-y', 'python3-perf-0:4.18.0-193.119.1.el8_2.x86_64']
12/02/2023 21:10:59 root [INFO]: Parsing response...
12/02/2023 21:10:59 root [INFO]: Running command ['dnf', 'list', 'installed']
12/02/2023 21:11:05 root [INFO]: Parsing response...
12/02/2023 21:11:05 root [INFO]: Running command ['dnf', 'module', 'list', '--enabled']
12/02/2023 21:11:14 root [INFO]: Parsing response...
12/02/2023 21:11:14 root [INFO]: Running command ['dnf', '--available', 'repoquery', '--queryformat', '%{NAME}.%{ARCH} %{EVR} %{REPONAME} %{BUILDTIME}']
12/02/2023 21:11:21 root [INFO]: Parsing response...
12/02/2023 21:11:53 root [INFO]: Running command ['rpm', '-qa', '--queryformat', '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{INSTALLTIME}\n']
12/02/2023 21:11:54 root [INFO]: Parsing response...
12/02/2023 21:11:57 root [INFO]: Installed count: 392, Installed Other count: 675, Failed count: 0, Not Applicable count: 7766, Missing count: 2, Installed Rejected count: 0
12/02/2023 21:11:57 root [INFO]: Retrying Install for Packages: ['ansible-0:7.2.0-1.el8.noarch', 'python3-paramiko-0:2.12.0-1.el8.noarch']
12/02/2023 21:11:57 root [INFO]: Running command ['dnf', 'install', '-y', 'ansible-0:7.2.0-1.el8.noarch']
12/02/2023 21:12:04 root [INFO]: Parsing response...
12/02/2023 21:12:04 root [WARNING]: Failed to install package: ansible-0:7.2.0-1.el8.noarch
12/02/2023 21:12:04 root [INFO]: Running command ['dnf', 'install', '-y', 'python3-paramiko-0:2.12.0-1.el8.noarch']
12/02/2023 21:12:13 root [INFO]: Parsing response...
12/02/2023 21:12:13 root [WARNING]: Failed to install package: python3-paramiko-0:2.12.0-1.el8.noarch
12/02/2023 21:12:13 root [INFO]: Running command ['dnf', 'list', 'installed']
12/02/2023 21:12:18 root [INFO]: Parsing response...
12/02/2023 21:12:18 root [INFO]: Running command ['dnf', 'module', 'list', '--enabled']
12/02/2023 21:12:26 root [INFO]: Parsing response...
12/02/2023 21:12:26 root [INFO]: Running command ['dnf', '--available', 'repoquery', '--queryformat', '%{NAME}.%{ARCH} %{EVR} %{REPONAME} %{BUILDTIME}']
12/02/2023 21:12:34 root [INFO]: Parsing response...

12/02/2023 21:13:06 root [INFO]: Running command ['rpm', '-qa', '--queryformat', '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{INSTALLTIME}\n']
12/02/2023 21:13:07 root [INFO]: Parsing response...
12/02/2023 21:13:09 root [INFO]: Installed count: 392, Installed Other count: 675, Failed count: 0, Not Applicable count: 7766, Missing count: 2, Installed Rejected count: 0
12/02/2023 21:13:09 root [INFO]:
Package compliance initialized with instance ID:InstanceID,
baseline ID: pbID, snapshot ID: 7d874638, patch group: ,
start time: 2023-12-03 02:04:22.662876, end time: 2023-12-03 02:13:09.752345, upload NA compliance: False,
install override list path: , execution id: CMDID

12/02/2023 21:13:09 root [INFO]: Instance is Non-Compliant with severity of INFORMATIONAL

Ok, let’s try to install ansible and see what it says:
[root@ServerNameCMDID]# dnf install --assumeno ansible-0:7.2.0-1.el8.noarch
Updating Subscription Management repositories.
This system is registered to Red Hat Subscription Management, but is not receiving updates. You can use subscription-manager to assign subscriptions.
Red Hat Satellite Client 6 for RHEL 8 x86_64 (RPMs) 95 kB/s | 3.8 kB 00:00
Red Hat Satellite Client 6 for RHEL 8 x86_64 - Update Services for Server Solutions (RPMs) 102 kB/s | 3.8 kB 00:00
epel8 91 kB/s | 2.3 kB 00:00
spp-el8 67 kB/s | 1.7 kB 00:00
Red Hat Ansible Engine 2.9 for RHEL 8 x86_64 (RPMs) 105 kB/s | 4.0 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS - Update Services for Server Solutions (RPMs) 106 kB/s | 4.1 kB 00:00
centrify-el8 79 kB/s | 2.0 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - High Availability - Update Services for Server Solutions (RPMs) 106 kB/s | 4.0 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream - Update Services for Server Solutions (RPMs) 124 kB/s | 4.5 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - Server Solutions - Update Services for Server Solutions (RPMs) 107 kB/s | 4.0 kB 00:00
Splunk OpenTelemetry Collector for x86_64 69 kB/s | 1.7 kB 00:00
ms-el8 62 kB/s | 1.5 kB 00:00
Error:
Problem: conflicting requests
- nothing provides /usr/bin/python3.11 needed by ansible-7.2.0-1.el8.noarch
- nothing provides python(abi) = 3.11 needed by ansible-7.2.0-1.el8.noarch
- nothing provides python3.11dist(ansible-core) >= 2.14.2 needed by ansible-7.2.0-1.el8.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

So, this is saying that ansible-0:7.2.0-1.el8.noarch requires /usr/bin/python3.11
This server is RHEL8.2 – this version of the OS didn’t have python3.11, it maxes out at python3.8. We would have to patch the OS to RHEL8.4 to get python3.11

So it looks like the issue for this server is that the patching is trying to apply patches that it shouldn’t.

Start with the captured output at /var/lib/amazon/ssm/InstanceID/document/orchestration/CMDID/runDocument/InstallNoReboot/InstallNoReboot/awsrunShellScript/PatchLinux/stdout and compare the listed dnf and rpm commands with the RHEL 8.2 package repositories. Reproduce the ansible dependency failure using dnf install --assumeno and trace the SSM patch-install entry point. Done means identifying why unavailable packages are selected and preventing the failed patch attempts from stopping compliance or reboot handling.

Written by the indexing model from the issue text.