[Bug]: Cannot Upgrade Salt Minion Using Salt

What happened?

Since Salt moved from Python 3.10 to Python 3.11, upgrading the minion from 3006.26 to 3006.27 fails midstream with a ModuleNotFoundError (full stack below). This, combined with the fact that the service restart portion of the postinst script was removed (https://github.com/saltstack/salt/commit/6f010915ba45355cdaf2c8de5ef775f67c34ab43#diff-aa664d27d8014d08bcf8241f8ca014945a98480389c07297ad794eafb55dbef6) has resulted in no clean way to upgrade Salt minions using Salt itself.

The underlying problem is that the package upgrade removes the /opt/saltstack/salt/lib/python3.10 module directory while the minion is running resulting in the minion crashing.

sudo salt-call -l debug state.sls updatesalt
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/ca_bundle.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/ca_bundle.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/master_address_do_not_change.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/master_address_do_not_change.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/s3.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/s3.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: testminion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/ca_bundle.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/ca_bundle.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/master_address_do_not_change.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/master_address_do_not_change.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/s3.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/s3.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: testminion
[DEBUG   ] Grains cache last modified 13341 seconds ago and cache expiration is set to 300. Grains cache expired. Refreshing.
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/ca_bundle.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/ca_bundle.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/master_address_do_not_change.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/master_address_do_not_change.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/s3.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/s3.conf
[DEBUG   ] The functions from module 'install_date' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'rh' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/zfs.py'>
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[INFO    ] Retrieved install_date from file
[DEBUG   ] in_dmz: [hostname=testminion, ipaddr=10.0.0.1]
[DEBUG   ] Loading static grains from /etc/salt/grains
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded zfs.is_supported
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] "salt" Not an IP address? Assuming it is a hostname.
[DEBUG   ] Master URI: tcp://172.24.55.8:4506
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'testminion', 'tcp://10.0.0.2:4506', '1730827032.201436')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (9810)
[DEBUG   ] Setting zmq_reconnect_ivl to '9810ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt._get_key_with_evict: Loading private key
[DEBUG   ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_23cf32f8d4_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_23cf32f8d4_pull.ipc
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] salt.crypt.verify_signature: Loading public key
[DEBUG   ] <salt.crypt.AsyncAuth object at 0x7fa0f0d3ca60> Got new master aes key.
[DEBUG   ] Closing AsyncReqChannel instance
[DEBUG   ] Connecting the Minion to the Master publish port, using the URI: tcp://10.0.0.2:4505
[DEBUG   ] <salt.transport.zeromq.PublishClient object at 0x7fa0f0d3cc70> connecting to tcp://10.0.0.2:4505
[DEBUG   ] Sending event: tag = __master_connected; data = {'master': 'salt', '_stamp': '2026-07-14T14:42:45.467015'}
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'testminion', 'tcp://10.0.0.2:4506', '1730827032.201436')
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Closing AsyncReqChannel instance
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded state.sls
[DEBUG   ] The functions from module 'direct_call' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.module.grains' from '/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/grains.py'>
[DEBUG   ] The functions from module 'grains' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'testminion', 'tcp://10.0.0.2:4506', '1730827032.201436')
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Gathering pillar data for state run
[DEBUG   ] Finished gathering pillar data for state run
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] In saltenv 'base', looking at rel_path 'updatesalt.sls' to resolve 'salt://updatesalt.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/updatesalt.sls' to resolve 'salt://updatesalt.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/updatesalt.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/base']
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/updatesalt.sls' using 'jinja' renderer: 0.004630088806152344
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/updatesalt.sls' using 'yaml' renderer: 0.0030698776245117188
[DEBUG   ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] The functions from module 'file' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded file.managed
[DEBUG   ] The functions from module 'pkg' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded pkg.install
[DEBUG   ] The functions from module 'pkg' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded pkg.installed
[DEBUG   ] LazyLoaded pkg.ex_mod_init
[INFO    ] Running state [salt-minion] at time 09:42:47.492196
[DEBUG   ] LazyLoaded pkg.resolve_capabilities
[DEBUG   ] The functions from module 'cmd' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded cmd.run_stdout
[INFO    ] Executing command dpkg-query in directory '/root'
[DEBUG   ] The functions from module 'pkg_resource' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded pkg_resource.add_pkg
[INFO    ] Executing command dpkg in directory '/root'
[DEBUG   ] LazyLoaded pkg.check_db
[INFO    ] Executing command apt-get in directory '/root'
[INFO    ] Executing command apt-cache in directory '/root'
[INFO    ] Executing command dpkg in directory '/root'
[DEBUG   ] LazyLoaded pkg.check_extra_requirements
[DEBUG   ] LazyLoaded pkg.version_clean
[INFO    ] Executing command dpkg in directory '/root'
[DEBUG   ] Current version (['3006.26']) did not match desired version specification (3006.27), adding to installation targets
[INFO    ] Running state [/usr/sbin/policy-rc.d] at time 09:42:50.195549
[INFO    ] Executing state file.managed for [/usr/sbin/policy-rc.d]
[DEBUG   ] The functions from module 'file' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded file.user_to_uid
[INFO    ] File changed:
New file
[INFO    ] Completed state [/usr/sbin/policy-rc.d] at time 09:42:50.204006 (duration_in_ms=8.457)
[INFO    ] Running state [salt-minion] at time 09:42:50.204360
[INFO    ] Executing state pkg.installed for [salt-minion]
[DEBUG   ] LazyLoaded pkg.resolve_capabilities
[INFO    ] Executing command dpkg in directory '/root'
[DEBUG   ] LazyLoaded pkg.check_db
[INFO    ] Executing command apt-cache in directory '/root'
[INFO    ] Executing command dpkg in directory '/root'
[DEBUG   ] LazyLoaded pkg.check_extra_requirements
[DEBUG   ] LazyLoaded pkg.version_clean
[INFO    ] Executing command dpkg in directory '/root'
[DEBUG   ] Current version (['3006.26']) did not match desired version specification (3006.27), adding to installation targets
[INFO    ] Executing command dpkg in directory '/root'
[INFO    ] Executing command dpkg in directory '/root'
[INFO    ] Executing command dpkg in directory '/root'
[INFO    ] Executing command dpkg in directory '/root'
[INFO    ] Executing command systemd-run in directory '/root'
[INFO    ] Executing command dpkg-query in directory '/root'
[DEBUG   ] LazyLoaded pkg.version_clean
[INFO    ] Made the following changes:
'salt-minion' changed from '3006.26' to '3006.27'
'salt-common' changed from '3006.26' to '3006.27'

[DEBUG   ] Refreshing modules...
[DEBUG   ] Closing AsyncReqChannel instance
[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
ModuleNotFoundError: spec not found for the module 'site'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 12, in <module>
    sys.exit(salt_call())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 465, in salt_call
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/call.py", line 113, in run
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/caller.py", line 95, in run
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/caller.py", line 200, in call
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 174, in __call__
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1375, in run
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1390, in _run_as
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/executors/direct_call.py", line 10, in execute
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 174, in __call__
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1375, in run
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1390, in _run_as
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 1626, in sls
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 3632, in call_high
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2805, in call_chunks
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 3345, in call_chunk
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/decorators/state.py", line 44, in _func
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2531, in call
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 1412, in check_refresh
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 1360, in module_refresh
  File "/opt/saltstack/salt/lib/python3.10/importlib/__init__.py", line 168, in reload
ModuleNotFoundError: spec not found for the module 'site'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 12, in <module>
    sys.exit(salt_call())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 465, in salt_call
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/call.py", line 113, in run
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/caller.py", line 95, in run
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/caller.py", line 200, in call
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 174, in __call__
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1375, in run
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1390, in _run_as
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/executors/direct_call.py", line 10, in execute
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 174, in __call__
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1375, in run
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1390, in _run_as
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 1626, in sls
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 3632, in call_high
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2805, in call_chunks
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 3345, in call_chunk
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/decorators/state.py", line 44, in _func
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2531, in call
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 1412, in check_refresh
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 1360, in module_refresh
  File "/opt/saltstack/salt/lib/python3.10/importlib/__init__.py", line 168, in reload
ModuleNotFoundError: spec not found for the module 'site'

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-22.04

salt --versions-report output

Salt Version:
          Salt: 3006.27

Python Version:
        Python: 3.11.15 (main, Jun 29 2026, 22:21:49) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 47.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.50
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.16
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5

System Versions:
          dist: ubuntu 22.04.5 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-185-generic
        system: Linux
       version: Ubuntu 22.04.5 jammy

Start by reproducing the reported salt-call -l debug state.sls updatesalt upgrade from 3006.26 to 3006.27 and inspect the package postinst behavior referenced by commit 6f010915. Trace what happens when /opt/saltstack/salt/lib/python3.10 is removed while the minion runs. Done means the minion upgrades without the ModuleNotFoundError and remains operational after the package upgrade.