CentOS 8 salt-minion returns non-zero exit code when using pkg state module to install appstreams

Description of Issue

A CentOS 8 salt minion returns a non-zero exit code when attempting to install the '@idm:client' appstream module using a salt state using pkg.installed. However when inspecting the minion's configuration afterwards, it does appear that the module packages are installed. Removing the module using pkg.removed claims the module is not installed.

I also tested this with the '@varnish' module and the results are identical.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

[root@repo staging]# cat authentication.sls
install_freeipa_client:
pkg.installed:
- name: '@idm:client'
- onlyif:
- test -f /etc/yum.repos.d/local.repo

[root@repo staging]# cat clean.sls
remove_ipa_client:
pkg.removed:
- name: '@idm:client'

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

1. Run the staging.authentication state to install the '@idm:client' appstream module.

root@repo staging]# salt '*' state.apply staging.authentication test=False
netboxsalt.private.danlab:

      ID: install_freeipa_client
Function: pkg.installed
    Name: @idm:client
  Result: False
 Comment: The following packages failed to install/update: @idm:client
 Started: 10:46:34.319663
Duration: 51514.05 ms
 Changes:
          ----------
          augeas-libs:
              ----------
              new:
                  1.12.0-5.el8
              old:
          autofs:
              ----------
              new:
                  1:5.1.4-43.el8
              old:
          certmonger:
              ----------
              new:
                  0.79.7-15.el8
              old:
          gssproxy:
              ----------
              new:
                  0.8.0-16.el8
              old:
          ipa-client:
              ----------
              new:
                  4.8.7-12.module_el8.3.0+514+e7703106
              old:
          ipa-client-common:
              ----------
              new:
                  4.8.7-12.module_el8.3.0+514+e7703106
              old:
          ipa-common:
              ----------
              new:
                  4.8.7-12.module_el8.3.0+514+e7703106
              old:
          ipa-selinux:
              ----------
              new:
                  4.8.7-12.module_el8.3.0+514+e7703106
              old:
          keyutils:
              ----------
              new:
                  1.5.10-6.el8
              old:
          krb5-workstation:
              ----------
              new:
                  1.18.2-5.el8
              old:
          libkadm5:
              ----------
              new:
                  1.18.2-5.el8
              old:
          libsss_simpleifp:
              ----------
              new:
                  2.3.0-9.el8
              old:
          libverto-libevent:
              ----------
              new:
                  0.3.0-5.el8
              old:
          nfs-utils:
              ----------
              new:
                  1:2.3.3-35.el8
              old:
          nss-tools:
              ----------
              new:
                  3.53.1-17.el8_3
              old:
          python3-asn1crypto:
              ----------
              new:
                  0.24.0-3.el8
              old:
          python3-augeas:
              ----------
              new:
                  0.5.0-12.el8
              old:
          python3-cffi:
              ----------
              new:
                  1.11.5-5.el8
              old:
          python3-cryptography:
              ----------
              new:
                  2.3-3.el8
              old:
          python3-dns:
              ----------
              new:
                  1.15.0-10.el8
              old:
          python3-gssapi:
              ----------
              new:
                  1.5.1-5.el8
              old:
          python3-ipaclient:
              ----------
              new:
                  4.8.7-12.module_el8.3.0+514+e7703106
              old:
          python3-ipalib:
              ----------
              new:
                  4.8.7-12.module_el8.3.0+514+e7703106
              old:
          python3-jwcrypto:
              ----------
              new:
                  0.5.0-1.module_el8.3.0+481+b9a999c3
              old:
          python3-ldap:
              ----------
              new:
                  3.1.0-5.el8
              old:
          python3-libipa_hbac:
              ----------
              new:
                  2.3.0-9.el8
              old:
          python3-netaddr:
              ----------
              new:
                  0.7.19-8.el8
              old:
          python3-netifaces:
              ----------
              new:
                  0.10.6-4.el8
              old:
          python3-pyasn1:
              ----------
              new:
                  0.3.7-6.el8
              old:
          python3-pyasn1-modules:
              ----------
              new:
                  0.3.7-6.el8
              old:
          python3-pycparser:
              ----------
              new:
                  2.14-14.el8
              old:
          python3-pyusb:
              ----------
              new:
                  1.0.0-9.module_el8.3.0+481+b9a999c3
              old:
          python3-qrcode-core:
              ----------
              new:
                  5.1-12.module_el8.3.0+481+b9a999c3
              old:
          python3-sss:
              ----------
              new:
                  2.3.0-9.el8
              old:
          python3-sss-murmur:
              ----------
              new:
                  2.3.0-9.el8
              old:
          python3-yubico:
              ----------
              new:
                  1.3.2-9.module_el8.3.0+514+e7703106
              old:
          rpcbind:
              ----------
              new:
                  1.2.5-7.el8
              old:
          sssd-dbus:
              ----------
              new:
                  2.3.0-9.el8
              old:
          sssd-tools:
              ----------
              new:
                  2.3.0-9.el8
              old:
          xmlrpc-c:
              ----------
              new:
                  1.51.0-5.el8
              old:
          xmlrpc-c-client:
              ----------
              new:
                  1.51.0-5.el8
              old:

Summary for netboxsalt.private.danlab

Succeeded: 0 (changed=1)
Failed: 1

Total states run: 1
Total run time: 51.514 s
ERROR: Minions returned with non-zero exit code

NOTE: Unlike the above, running salt '*' pkg.install '@idm:client' does not return an error.

2. Running a cleaning state to remove the module/packages afterwards runs successfully, but claims the packages are already absent.

[root@repo staging]# salt '*' state.apply staging.clean test=False
netboxsalt.private.danlab:

      ID: remove_ipa_client
Function: pkg.removed
    Name: @idm:client
  Result: True
 Comment: All specified packages are already absent
 Started: 10:51:10.399446
Duration: 927.278 ms
 Changes:

Summary for netboxsalt.private.danlab

Succeeded: 1
Failed: 0

Total states run: 1
Total run time: 927.278 ms

Looking at the minion state afterwards the 41 packages are still installed.
NOTE: When running the equivalent dnf module remove command from CLI, I noted that the '@' must be absent, e.g. sudo dnf module remove idm:client. However, this still yields the same result if the SLS state uses 'idm:client' instead of '@idm:client'.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Master:
Salt Version:
Salt: 3003

Dependency Versions:
cffi: 1.11.5
cherrypy: 18.4.0
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Aug 24 2020, 17:57:11)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 19.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist: centos 8
locale: UTF-8
machine: x86_64
release: 4.18.0-240.22.1.el8_3.x86_64
system: Linux
version: CentOS Linux 8

Minion:
Salt Version:
Salt: 3003

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Aug 24 2020, 17:57:11)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 19.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist: centos 8
locale: UTF-8
machine: x86_64
release: 4.18.0-240.22.1.el8_3.x86_64
system: Linux
version: CentOS Linux 8

Start with authentication.sls and clean.sls, then trace the pkg.installed and pkg.removed state handling for CentOS 8 AppStream module names such as @idm:client and @varnish. Reproduce the state commands against the reported Salt 3003 environment and compare them with the working pkg.install CLI behavior; done means installation and removal report accurate results and exit status.