[Bug]: Memory leak in salt-master 3006.27

What happened?

We have several salt masters, running on Ubuntu 22.04. They all had version 3006.13 as it was stable.
We tried to upgrade them, first the test one, then all others, but we couldn't find a newer stable version until 3006.27.
We tried 3006.16 and 3006.19 and even on test salt master with just a few salt minion there was a memory leak in salt-api.

And finally 3006.27 seemed good, no memory leaks in salt-api, so we upgraded all our salt masters to 3006.27.
All of salt-masters are doing well except one. The only difference with the other salt masters is that this one has a lot of minions that are not available at the moment.

Memory utilization looks like this:

When it gets to the peak:

Some time before salt-master goes completely crazy it had the errors like this (several times):

2026-07-27 07:31:56,432 [salt.master      :1956][ERROR   ][2425] Error in function _return:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/master.py", line 1950, in run_func
    ret = getattr(self, func)(load)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/master.py", line 1743, in _return
    salt.utils.job.store_job(
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/job.py", line 31, in store_job
    return _store_job(opts, load, event, mminion, endtime=endtime)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/job.py", line 91, in _store_job
    event.fire_event(
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/event.py", line 883, in fire_event
    self.pusher.send(msg)
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 153, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 169, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
              ^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/transport/ipc.py", line 445, in send
    yield self.stream.write(pack)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/iostream.py", line 409, in write
    self._add_io_state(self.io_loop.WRITE)
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/iostream.py", line 975, in _add_io_state
    self.io_loop.update_handler(self.fileno(), self._state)
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/ioloop.py", line 734, in update_handler
    self._impl.modify(fd, events | self.ERROR)
FileNotFoundError: [Errno 2] No such file or directory

Then this one:

2026-07-27 08:00:59,982 [salt.client      :1916][ERROR   ][648013] Message timed out
2026-07-27 08:01:00,024 [salt._logging.impl:1101][ERROR   ][648013] An un-handled exception was caught by Salt's global exception handler:
SaltClientError: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can 
then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 1914, in pub
    payload = channel.send(payload_kwargs, timeout=timeout)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 153, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 169, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/channel/client.py", line 380, in send
    ret = yield self._uncrypted_transfer(load, timeout=timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/channel/client.py", line 354, in _uncrypted_transfer
    ret = yield self.transport.send(self._package_load(load), timeout=timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/transport/zeromq.py", line 1404, in send
    ret = yield self.message_client.send(load, timeout=timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/transport/zeromq.py", line 923, in send
    recv = yield future
           ^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
salt.exceptions.SaltReqTimeoutError: Message timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 386, in run_job
    pub_data = self.pub(
               ^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 1917, in pub
    raise SaltReqTimeoutError(
salt.exceptions.SaltReqTimeoutError: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/salt", line 12, in <module>
    sys.exit(salt_main())
             ^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/scripts.py", line 552, in salt_main
    client.run()
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/cli/salt.py", line 55, in run
    self._run_batch()
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/cli/salt.py", line 291, in _run_batch
    for res, job_retcode in batch.run():
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/cli/batch.py", line 239, in run
    part = next(queue)
           ^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 986, in cmd_iter_no_block
    for fn_ret in self.get_iter_returns(
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 1270, in get_iter_returns
    jinfo = self.gather_job_info(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 294, in gather_job_info
    pub_data = self.run_job(
               ^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 408, in run_job
    raise SaltClientError(general_exception)
salt.exceptions.SaltClientError: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.

Salt master gets tasks to do salt deployments from salt-api, the queue grows and grows until it eats all the memory.

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-22.04

salt --versions-report output

Salt Version:
          Salt: 3006.27
 
Python Version:
        Python: 3.11.15 (main, Jun 29 2026, 22:21:49) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 47.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.50
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.16
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5
 
System Versions:
          dist: ubuntu 22.04.5 jammy
        locale: utf-8
       machine: x86_64
       release: 6.8.0-124-generic
        system: Linux
       version: Ubuntu 22.04.5 jammy