[BUG] ValueError: Incorrect decryption.

Description
We try to install salt on Almalinux 9 and run a first highstate on the salt master himself to install the base package "htop". Nothing special so far.

This is a clean install.

Master and Minion are running and config return any error.

Command: salt 'salt*' state.highstate -l debug

Output:

[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Missing configuration file: /root/.saltrc
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Closing AsyncReqChannel instance
[DEBUG   ] The functions from module 'local_cache' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/94/3a769634ca0f986dbc32fcbc7339829305868004dc1f37f0ba5d2c6f30956f/.minions.p
[DEBUG   ] get_iter_returns for jid 20221115081738180606 sent to {'salt.example.com'} will timeout at 09:27:38.237405
[DEBUG   ] jid 20221115081738180606 return from salt.example.com
[DEBUG   ] return event: {'salt.example.com': {'ret': 'The minion function caused an exception: Traceback (most recent call last):\n  File "salt/minion.py", line 1935, in _thread_return\n    return_data = minion_instance._execute_job_function(\n  File "salt/minion.py", line 1894, in _execute_job_function\n    return_data = self.executors[fname](opts, data, func, args, kwargs)\n  File "salt/loader/lazy.py", line 149, in __call__\n    return self.loader.run(run_func, *args, **kwargs)\n  File "salt/loader/lazy.py", line 1228, in run\n    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)\n  File "salt/loader/lazy.py", line 1243, in _run_as\n    return _func_or_method(*args, **kwargs)\n  File "/opt/saltstack/salt/run/salt/executors/direct_call.py", line 10, in execute\n    return func(*args, **kwargs)\n  File "salt/loader/lazy.py", line 149, in __call__\n    return self.loader.run(run_func, *args, **kwargs)\n  File "salt/loader/lazy.py", line 1228, in run\n    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)\n  File "salt/loader/lazy.py", line 1243, in _run_as\n    return _func_or_method(*args, **kwargs)\n  File "/opt/saltstack/salt/run/salt/modules/state.py", line 1102, in highstate\n    st_ = salt.state.HighState(\n  File "salt/state.py", line 4769, in __init__\n    self.state = State(\n  File "salt/state.py", line 768, in __init__\n    self.opts["pillar"] = self._gather_pillar()\n  File "salt/state.py", line 833, in _gather_pillar\n    return pillar.compile_pillar()\n  File "salt/pillar/__init__.py", line 353, in compile_pillar\n    ret_pillar = self.channel.crypted_transfer_decode_dictentry(\n  File "salt/utils/asynchronous.py", line 125, in wrap\n    raise exc_info[1].with_traceback(exc_info[2])\n  File "salt/utils/asynchronous.py", line 131, in _target\n    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))\n  File "salt/ext/tornado/ioloop.py", line 459, in run_sync\n    return future_cell[0].result()\n  File "salt/ext/tornado/concurrent.py", line 249, in result\n    raise_exc_info(self._exc_info)\n  File "<string>", line 4, in raise_exc_info\n  File "salt/ext/tornado/gen.py", line 1070, in run\n    yielded = self.gen.send(value)\n  File "salt/channel/client.py", line 187, in crypted_transfer_decode_dictentry\n    aes = cipher.decrypt(ret["key"])\n  File "Cryptodome/Cipher/PKCS1_OAEP.py", line 200, in decrypt\n    raise ValueError("Incorrect decryption.")\nValueError: Incorrect decryption.\n', 'out': 'nested', 'retcode': 1, 'jid': '20221115081738180606'}}
[DEBUG   ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded nested.output
salt.example.com:
    The minion function caused an exception: Traceback (most recent call last):
      File "salt/minion.py", line 1935, in _thread_return
        return_data = minion_instance._execute_job_function(
      File "salt/minion.py", line 1894, in _execute_job_function
        return_data = self.executors[fname](opts, data, func, args, kwargs)
      File "salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "salt/loader/lazy.py", line 1228, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "salt/loader/lazy.py", line 1243, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/opt/saltstack/salt/run/salt/executors/direct_call.py", line 10, in execute
        return func(*args, **kwargs)
      File "salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "salt/loader/lazy.py", line 1228, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "salt/loader/lazy.py", line 1243, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/opt/saltstack/salt/run/salt/modules/state.py", line 1102, in highstate
        st_ = salt.state.HighState(
      File "salt/state.py", line 4769, in __init__
        self.state = State(
      File "salt/state.py", line 768, in __init__
        self.opts["pillar"] = self._gather_pillar()
      File "salt/state.py", line 833, in _gather_pillar
        return pillar.compile_pillar()
      File "salt/pillar/__init__.py", line 353, in compile_pillar
        ret_pillar = self.channel.crypted_transfer_decode_dictentry(
      File "salt/utils/asynchronous.py", line 125, in wrap
        raise exc_info[1].with_traceback(exc_info[2])
      File "salt/utils/asynchronous.py", line 131, in _target
        result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
      File "salt/ext/tornado/ioloop.py", line 459, in run_sync
        return future_cell[0].result()
      File "salt/ext/tornado/concurrent.py", line 249, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 4, in raise_exc_info
      File "salt/ext/tornado/gen.py", line 1070, in run
        yielded = self.gen.send(value)
      File "salt/channel/client.py", line 187, in crypted_transfer_decode_dictentry
        aes = cipher.decrypt(ret["key"])
      File "Cryptodome/Cipher/PKCS1_OAEP.py", line 200, in decrypt
        raise ValueError("Incorrect decryption.")
    ValueError: Incorrect decryption.
[DEBUG   ] jid 20221115081738180606 found all minions {'salt.example.com'}
[DEBUG   ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code
[DEBUG   ] Restoring sys.argv to: ['/opt/saltstack/salt/run/run', 'salt*', 'state.highstate', '-l', 'debug']

Setup

We run the salt setup on a promox VM. Other setups with Ubuntu (20.04 and 22.04) is running fine.

we use the "repo.saltproject.io/salt/py3/redhat/9/x86_64/latest" repo and every is updated to the last version.

Steps to Reproduce the behavior

Expected behavior
installing the given package

Versions Report

base-packages:
  pkg.installed:
    - pkgs:
      - htop

Does anybody know whats going wrong here?

The failure is reported at salt/channel/client.py:187 during pillar compilation from salt/pillar/init.py:353. Start by reproducing salt 'salt*' state.highstate -l debug on the reported AlmaLinux 9 and compare it with the working Ubuntu setups. Done means the highstate completes without the decryption exception and installs the htop package.